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Abstract 

The  Military  Health  Service  (MHS),  feeling  growing  pressure  to  keep  costs  down  and  quality  up, 
must  be  able  to  strike  a  cost-quality  balance.  Success  of  initiatives  like  the  Medicare  Subvention 
Demonstration  Project  is  dependent  on  the  ability  to  strike  such  a  balance — and  the  ability  to 
monitor  and  report  it.  MHS’s  lack  of  Medicare  HMO  experience  necessitates  the  development 
and  adoption  of  reliable  performance  measurement  systems.  Project  managers  must  align 
organizational  mission  and  health  plan  performance.  Current  military  measurement  initiatives — 
the  Air  Force  Medical  Service  Performance  Measurement  Tool  (AFMS  PMT)  and  the  MHS 
Performance  Report  Card — compare  well  to  major  civilian  initiatives — the  Consortium 
Research  on  Indicators  of  System  Performance  (CRISP),  the  Comprehensive  Healthcare 
Analysis  and  Management  Program,  the  Health  Plan  Employer  Data  and  Information  Set 
(HEDIS)  and  two  related  research  projects  from  the  Health  Care  Advisory  Board.  MHS 
performance  measurement  systems  do,  however,  require  modification  to  be  useful  for  Medicare 
health  plan  management.  Emphasis  on  Medicare  population-specific  indicators  must  be 
included — indicators  that  stress  the  importance  of  long-term  versus  short-term  measurement, 
wellness  and  prevention  versus  episodic  treatment  and  reaction,  and  primary  care  investment 
versus  specialty  care  expenditure.  The  uniqueness  of  an  MHS  Medicare  Subvention  Project 
environment  make  it  unlikely  that  any  one  of  the  initiatives  studied  would  be  sufficient  in 
Medicare  health  plan  performance  measurement.  However,  the  combination  of  three  models 
(AFMS  PMT,  CRISP,  and  HEDIS)  holds  promise  as  a  model  Medicare  Subvention 
Demonstration  Project  performance  measurement  system.  AFMS  PMT  and  CRISP  provide  the 
means  to  monitor  health  plan  performance  and  HEDIS  provides  the  means  to  benchmark  and 
report  health  plan  performance  improvement. 
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Introduction 

Twenty  years  ago,  managed  care  was  considered  an  emerging  set  of  techniques  that  could 
be  used  to  manage  health  care  costs  by  influencing  provider  decisions  (Gray  &  Field,  1989). 
What  was  then  considered  an  optional  activity  aimed  at  revenue  enhancement  and  cost  reduction 
has  since  become  a  dominating  force.  Managed  care  principles  are  being  employed  in  varying 
degrees  across  the  entire  health  care  spectrum  (Fox,  1996,  p.  3).  Management  of  military 
healthcare  delivery  is  no  exception. 

As  with  civilian  health  care  organizations,  the  Military  Health  Service  (MHS),  formerly 
Military  Health  Service  System  (MHSS),  is  feeling  growing  pressure  to  keep  costs  down  and 
quality  up  (Grier,  1998,  p  41).  Over  the  last  ten  years,  the  number  of  military  treatment  facilities 
(MTFs)  providing  services  have  decreased  by  35  percent  while  corresponding  force  reductions 
account  for  only  a  nine  percent  decrease  in  eligible  beneficiaries  (Joseph,  1997b).  To  compound 
the  problem,  the  1998  defense  budget  was  almost  $200  million  less  than  expected.  Dr.  Ed 
Martin,  Acting  Assistant  Secretary  of  Defense  for  Health  Affairs,  remarked  that  “despite 
insufficient  funding,  the  MHS  must  live  within  the  appropriation  so  as  not  to  compromise  patient 
care  (Gillert,  1997).  In  such  an  environment,  striking  a  cost-quality  balance  that  maintains  access 
and  quality  yet  controls  cost  becomes  the  challenge.  Central  to  striking  this  balance  is  the  ability 
to  measure  health  plan  performance  (Konstvedt,  1996). 

Selecting  key  performance  indicators,  those  “few  areas  of  activity  necessary  for  a 
particular  organization  to  achieve  its  purpose”  (Rockart  as  cited  in  Duncan,  Ginter,  &  Swayne, 
1995,  p.  198),  enables  health  plan  executives  to  gauge  performance.  The  Joint  Commission  on 
Accreditation  of  Healthcare  Organizations  (JCAHO)  (n.d./1998b)  echoes  the  importance  of 


performance  measurement: 
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As  quality  of  care  becomes  an  even  more  visible  public  issue,  increased  attention  is  being 
given  to  the  effectiveness  of  current  quality  oversight  processes  and  systems.  The 
growing  demand  for  objective,  comparative  information  about  the  performance  of  health 
care  organizations  has  created  a  need  for  data-driven  evaluation  processes,  (p.  1) 

As  military  healthcare  evolves,  fewer  and  fewer  differences  exist  between  MHS  and 
civilian  health  systems.  The  adoption  of  enrollment-based  capitation  (EBC)  by  MHS  facilities, 
starting  Fiscal  Year  1998,  significantly  “grayed”  the  distinction  between  the  two.  EBC  divides 
the  annual  Defense  Health  Plan  (DHP)  appropriation  on  a  per  capita  basis  and  “provides  the 
proper  incentives  to  encourage  every  commander,  provider,  and  decision  maker  to  be  fully 
accountable  for  delivering  high-quality,  cost-effective  healthcare  services  to  [MHS] 
beneficiaries”  (Martin,  1997,  p.  1).  Such  a  goal  must  be  supported  by  a  robust  performance 
measurement  system. 

The  need  for  health  plan  performance  measurement  forms  the  heart  of  this  study.  The 
purpose  of  this  study  is  to  determine  if  current  MHS  performance  measurement  systems  are 
adequate  to  evaluate  health  plan  effectiveness.  “Health  plan”  will  be  operationally  defined  as  an 
“arrangement  by  which  persons,  including  dependents  or  spouses,  covered  or  making  application 
to  be  covered  under  this  pool,  have  access  to  hospital  and  medical  benefits  or  reimbursement” 
(Washington  State  Legislature,  n.d./1998).  In  the  context  of  the  MHS,  a  Medicare  Subvention 
Demonstration  Project  at  a  particular  MTF  may  be  considered  a  health  plan. 

Medicare  subvention,  also  called  TRICARE  Senior  Prime,  extends  health  maintenance 
organization  (HMO)  type  benefits  in  the  form  of  guaranteed  access  to  MTFs  and  primary  care 
managers  (PCMs)  to  Medicare-eligible,  retired  military  personnel,  their  family  members,  and 
survivors.  The  fiscal  year  1997  Defense  Authorization  Bill  (1997)  defines  Medicare  subvention 
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as  a  program  where  the  MHS  would  be  reimbursed  by  the  Health  Care  Financing  Administration 
(HCFA)  for  care  provided  to  Medicare-eligible  beneficiaries.  The  bounds  of  the  three-year 
project  are  twofold.  First,  the  MHS  must  continue  to  maintain  historical  levels  of  care,  or  level  of 
effort  (LOE),  provided  to  Medicare-eligible  beneficiaries.  The  second  provision  ensures  budget 
neutrality  to  the  overall  federal  program  for  additional  care  provided  (Joseph,  1997a).  This 
demonstration  project  makes  the  ability  to  accurately  measure  MHS  health  plan  performance 
more  critical  than  ever.  Successful  scoring  of  the  demonstration  project  is  central  to  a  win-win 
relationship  for  beneficiaries  and  the  federal  program. 

The  research  question,  framed  within  the  environment  of  Medicare  subvention,  is:  Are 
current  MHS  performance  measurement  systems  adequate  to  evaluate  Medicare 
Subvention  Demonstration  success?  Conclusions  will  be  drawn  from  a  comparison  of  current 
military  and  civilian  measurement  initiatives.  Primary  initiatives  to  be  compared  include:  (a)  the 
Air  Force  Medical  Service  Performance  Measurement  Tool  (Foster,  1998;  Air  Force  Medical 
Service,  1998),  (b)  the  MHS  Performance  Report  Card  (Health  Services  Analysis  and 
Measurement,  1997),  (c)  the  Consortium  Research  on  Indicators  of  System  Performance 
(Nerenz,  Zajac,  &  Rosman,  1993),  (d)  the  Comprehensive  Healthcare  Analysis  and  Management 
Program  (Mercer,  1997),  (e)  the  Health  Plan  Employer  Data  and  Information  Set  (National 
Committee  for  Quality  Assurance,  1997c;  1998),  and  (f)  two  related  research  projects  from  the 
Health  Care  Advisory  Board  (1995;  1996a). 

Limitations 

It  is  important  to  note  that  although  parallels  can  be  drawn  between  civilian  and  military 
performance  measurement  methodologies,  resulting  measures  may  not  be  reliably  compared — 
significant  differences  exist.  For  example,  provider  motivation  in  a  typical  civilian  organization 
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is  driven,  in  part,  by  monetary  factors.  The  vast  majority  of  physicians’  salaries  are  based  on 
work  done  (fee-for-service  models)  or  on  work  not  done  (capitation  models)  (Gapenski,  1996). 
Military  physicians,  however,  are  salaried  providers  whose  wage  is  not  subject  to  risk  pooling  or 
the  volume  of  work  produced.  While  typical  managed  care  elements,  such  as  utilization  and  case 
management,  are  used  in  military  organizations,  physicians  have  little  personal  incentive  for 
efficiency.  Efficiency  does  not  affect  their  incomes. 

Different  organizational  missions  also  contribute  to  the  difficulty  in  drawing  direct 
comparisons.  For-profit  and  not-for-profit  civilian  health  care  organizations  are  expected  to 
return  value  to  their  owners:  maximized  profits  and/or  expanded  services.  Both  require  excess 
revenue  in  order  to  accomplish  this  mission  (Griffith,  1995;  see  also  Shukla,  Pestian,  &  Clement, 
1997).  The  MHS  mission,  however,  centers  around  medical  readiness  which  “provides  health 
care  services  and  support  to  the  Armed  Forces,  in  all  operational  environments,  and  care  for 
other  authorized  beneficiaries”  (Office  of  the  Assistant  Secretary  of  Defense,  Health  Affairs 
[OSD  (HA)]  n.d./1998,  p.  5). 

The  Department  of  Defense  (DoD)  (1995)  Medical  Readiness  Strategic  Plan  (MRSP) 
defines  medical  readiness  as: 

the  ability  to  mobilize,  deploy  and  sustain  field  medical  services  and  support  for  any 
operation  requiring  military  services;  to  maintain  and  project  the  continuum  of  healthcare 
resources  required  to  provide  for  the  health  of  the  force;  and  to  operate  in  conjunction 
with  beneficiary  healthcare,  (p.  i) 

In  peacetime,  MHS  medical  systems  and  organizations  for  wartime  support  provide 
healthcare  in  much  the  same  manner  as  civilian  counterparts;  however,  recent  changes  in  U.  S. 
foreign  and  domestic  policy  significantly  impacts  the  Defense  medical  community.  These 


Performance  Measurement  5 


policies,  coupled  with  budget  and  personnel  reductions  in  the  armed  forces,  force  MHS  leaders 
to  place  more  emphasis  on  the  primary  mission:  medical  readiness  (DoD,  1995,  p.  6).  The  MHS 
must  strive  to  provide  all  support  required  within  annual  budgetary  constraints  (Hart  &  Connors. 
1996).  Since  potential  MHS  workload  exceeds  current  and  projected  capability,  little  room  exists 
for  the  accumulation  of  any  operating  surplus  (revenue). 

Another  possible  limitation  of  the  study  is  rooted  in  mandated  measurement  systems  for 
the  subvention  test.  DoD’s  Corporate  Executive  Information  System  (CEIS)  is  the  official  source 
of  demonstration  data  as  prescribed  by  the  OSD  (HA)  annotated  version  of  the  Code  of  Federal 
Regulations,  Title  42,  Volume  2,  Section  417.126  (HCFA,  1997).  The  limitation  lies  in  the  fact 
that,  although  CEIS  is  the  state-of-the-art  MHS  performance  measurement  system  (Corporate 
Executive  Information  System  Executive  Agent  [CEIS  (EA)],  1998),  DoD  lacks  experience  in 
managing  and  measuring  senior  healthcare  delivery.  Baseline  information  must  be 
retrospectively  extracted  from  legacy  information  systems  that  were  not  designed  to  capture 
Medicare-related  data.  The  ability  to  measure  demonstration  project  success  is  dependent  on 
validation  of  historical  data. 

Finally,  the  official  scoring  methodology  of  the  demonstration  project  is  a  potential 
limitation  of  this  study.  The  General  Accounting  Office  (GAO)  will  accomplish  the  actual 
project  scoring.  The  Comptroller  General  will  determine  the  extent,  if  any,  to  which  MHS  and 
HCFA  costs  have  increased  as  a  result  of  the  subvention  program  (Hefley,  1997,  p.  3);  however, 
the  criteria  used  in  the  cost  determination  has  not  been  provided  to  either  the  MHS  or  HCFA. 
Effective  project  management  is  highly  dependent  on  accurate  health  plan  performance 
measurement.  Early  reconciliation  between  performance  measurement  systems  used  by 
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demonstration  managers  and  those  employed  by  GAO  is  crucial  to  avoid  ambiguity  in  the 
evaluation  process. 

While  the  study  focuses  on  performance  measurement  data  elements,  or  indicators, 
versus  specific  data  values,  it  is  important  to  briefly  touch  on  inherent  limitations  of  existing 
MHS  data.  Even  the  most  up-to-date,  sophisticated  MHS  information  system,  CEIS,  is  only  as 
good  as  the  data  it  receives.  CEIS  receives  data  feeds  from  a  variety  of  component  legacy 
systems  (described  in  Appendix  B)  that  will  be  maintained  until  their  functionality  has  been 
incorporated  in  CEIS  and  CEIS  is  fully  deployed  (CEIS  (EA),  1998).  Unfortunately,  the  MHS 
has  historically  been  more  successful  in  providing  cost-efficient,  quality  health  care  than  it  has 
been  at  quantifying  and  reporting  on  its  health  care  delivery  process  (B.  J.  Kerr,  personal 
communication,  February,  1997).  Data  reported  through  legacy  systems  has  been  notoriously 
inaccurate.  In  fact,  Alan  Constantine  (1998)  from  the  Health  Program  Analysis  and  Evaluation 
Directorate  of  the  TRICARE  Management  Activity  (TMA)  recently  remarked  that  “data  quality 
borders  on  crisis.”  The  TMA  (1998)  stresses  that  “data  quality  is  paramount  and  MEPRS 
[Medical  Expense  and  Performance  Reporting  System]  data  are  key  to  accurate  cost 
determination  and  pricing”  (p.  31).  Significant  efforts  aimed  at  data  quality  must  precede 
meaningful  performance  comparison. 


Literature  Review 

An  overview  of  performance  measurement  in  the  healthcare  industry  is  better  understood 
in  the  frame  of  “how  did  we  get  here?”  (Mason  &  Dawson,  1998).  Table  1  provides  a  summary 
of  sentinel  themes  and  industry’s  response. 
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Table  1 


Healthcare  Response  and  Focus  to  Industry  Evolution 

Theme 

Response 

Planning  Focus 

Access 

Hill  Burton 

Facilities  planning 

Expand  capabilities 

Regional  Medical  Programs 
(RMP) 

Comprehensive  Health  Planning 
(CHP) 

Program  planning 

Excess  capacity, 
uncontrolled  cost 

Prospective  Payment  System 
(PPS) 

Shift  to  managed  care 

Strategic  Planning 

Medical  Management 
Disease  Management 
Demand  Management 

Clinical  performance 
Improvement 

Performance  based  planning 

Note.  From  “Competing  on  Quality  and  Clinical  Outcomes,”  by  S.  A.  Mason  and  S.  Dawson, 
1998,  Presentation  at  American  College  of  Healthcare  Executives  Congress  on  Healthcare 
Management,  March  2, 1998. 


Access  to  healthcare  was  a  major  issue  in  the  early  1940s.  Few  hospitals  had  been 
constructed  between  the  Depression  and  World  War  II,  manifesting  in  a  severe  shortage  of 
hospital  beds  across  the  nation.  The  government  responded  with  the  Hill  Burton  Act,  the 
Government’s  first  intervention  into  health  planning.  The  program  provided  assistance  to  states 
and  communities  in  the  form  of  both  planning  and  federal  matching  grants.  Hill  Burton 
effectively  solved  the  short-term  problem  of  access  by  assisting  in  the  construction  of  nearly  40 
percent  of  the  nation's  hospitals  (Mason  &  Dawson,  1998;  see  also  Williams  &  Torrens,  1993). 

The  next  major  theme  in  U.S.  healthcare  was  the  expansion  of  system  capabilities. 
Government’s  answer  was  Regional  Medical  Programs  (RMP)  and  Comprehensive  Health 
Planning  (CHP).  The  RMP  Act  was  a  federal  program  focusing  on  funding  and  planning  for 
detection  and  treatment  of  specific  health  problems  such  as  cancer  and  heart  disease.  CHP 
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represented  the  next  step  in  the  federal  planning  process.  CHP  was  developed  in  an  attempt  to  tie 
Medicare  and  Medicaid  funding  to  initiatives  that  encouraged  more  cost-effective  healthcare 
delivery.  Unfortunately,  the  effort  to  promote  voluntary  health  planning  through  a  consortium  of 
local,  state,  and  federal  agencies  failed  to  generate  any  measurable  increase  in  either  access  to 
care  or  quality  of  care  (Mason  &  Dawson,  1998;  see  also  Williams  &  Torrens,  1993). 

The  third  theme  became  apparent  in  the  early  1980s:  an  effort  to  attack  excess  capacity 
and  uncontrolled  cost.  Prospective  payment,  fixed  fees  based  on  diagnosis  rather  than  on  the 
procedure  performed,  announced  the  shift  from  fee-for-service  reimbursement  to  managed  care. 
Medicare  fee  schedules  were  devised  and  inpatient  costs  were  capped  according  to  diagnostic 
categories  called  diagnostic  related  groups  (DRGs).  With  Medicare,  the  largest  payor  in  the  U.S., 
taking  the  lead,  commercial  insurers  soon  followed  by  either  accepting  Medicare’s  fee  schedule 
or  by  devising  similar  payment  methodologies  (Mason  &  Dawson,  1998). 

Mason  and  Dawson  (1998)  maintain  that  healthcare’s  next  defining  theme  centers  around 
three  distinct  areas:  medical  management,  disease  management,  and  demand  management. 
Performance  based  measurement,  planning,  and  improvement  offers  healthcare  a  viable  strategy 
to  meet  the  challenge  of  improving  healthcare  organization  performance. 

Like  the  evolution  of  healthcare  themes  proposed  by  Mason  and  Dawson,  the  evolution 
of  healthcare  performance  measurement  can  be  gleaned  from  the  literature.  Current  trends  in 
performance  measurement  can  be  traced  back  to  healthcare’s  fee-for-service  roots. 

Prior  to  widespread  government  programs  like  Medicare  and  Medicaid,  “the  typical 
American  hospital  needed  little  more  than  simple  bookkeeping.  Once  a  year  a  decision  was  made 
to  raise  prices”  (Beck,  1980,  p.  1).  Accounting  was  a  necessary  evil;  it  did  not  contribute  directly 
to  the  hospital’s  mission  in  the  minds  of  many  administrators.  Needless  to  say,  in  the  period 
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from  the  first  Medicare  cost  report  in  1967  to  today,  the  importance  and  complexity  of 
performance  measurement  has  dramatically  increased  (Beck,  1980). 

As  the  healthcare  industry  evolved  during  the  past  three  decades,  measurement  efforts 
paralleled  other  industries.  Commonly  used  quantitative  measures  of  organizational 
performance,  traditional  financial  ratios  and  market  standing  became  important.  Ratios  such  as 
return  on  investment,  return  on  equity,  profit  margin,  debt  to  equity,  revenue  growth,  and  asset 
growth  provide  quick  and  easy  measures  to  evaluate  and  compare  organizations.  Additionally, 
overall  market  share,  share  of  the  target  market,  and  market  share  as  a  percentage  of  the  market 
leader  illustrate  an  organization’s  market  position  (Duncan,  Ginter,  and  Swayne,  1995,  p.  533). 

Peter  Drucker  (in  Duncan,  Ginter,  &  Swayne,  1995)  takes  organizational  performance 
measurement  a  step  further.  He  indicates  that  five  measures  can  describe  performance:  market 
standing,  innovative  performance,  liquidity  and  cash  flow,  profitability,  and  productivity.  It  is  in 
the  area  of  productivity  that  pre-prospective  payment  (i.e.  fee-for-service)  healthcare  began  to 
develop  industry-unique  measures. 

The  fee-for-service  healthcare  industry  measured  productivity  in  terms  such  as  patient 
admissions,  patient  visits,  and  total  surgical  procedures  performed.  Higher  workload  was 
rewarded  with  higher  reimbursement  (Kovner,  1995).  Skyrocketing  healthcare  costs  during  the 
1970s  and  1980s  were  partly  attributed  to  the  practice  of  “more  is  better,”  often  from  both  the 
patients’  and  physicians’  points-of-view  (Hodgetts  &  Cascio,  1983,  p.  299;  see  also  Vladeck, 
1997,  p.  4).  Due  to  its  newly  designed  reporting  requirements,  Medicare’s  influence  on  the 
market  during  that  period  significantly  changed  performance  measurement  (Siren  &  Laffel, 


1996). 
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Large  administrative  datasets  compiled  to  support  Medicare  claims  processing  opened 
the  door  for  investigators  to  analyze  practice  variations  and  other  related  phenomena.  The 
availability  of  claims  data  and  the  development  of  sophisticated  computer  systems,  combined 
with  increasing  pressure  to  contain  rising  health  care  expenditures,  gave  rise  to  performance 
measurement  as  it  is  known  today.  Structure,  process,  and  outcome  criteria,  developed  by 
Avedias  Donabedian  during  this  time,  can  still  be  found  at  the  heart  of  current  performance 
measurement  systems  (Iezzoni,  1997;  Siren  &  Laffel,  1996). 

Structural  performance  measures  focus  on  the  context  that  services  are  provided.  Typical 
structure  measures  include  board  certification  of  physicians,  licensure  of  facilities,  compliance 
with  safety  codes,  and  recordkeeping.  “Structure  measures  generally  do  not  offer  adequate 
specificity  to  differentiate  the  capabilities  of  providers  or  organizations  beyond  meeting 
minimum  standards”  (Siren  &  Laffel,  1996,  P.  402). 

On  the  other  hand,  process  measures  of  health  care  evaluate  the  way  care  is  provided. 
Process  measures  are  still  retrospective  and  examine  the  delivery  system  versus  the  product — 
healthcare  services.  Examples  of  common  process  measures  are  the  number  of  referrals  made 
out  of  network,  health-screening  rates,  follow-up  rates  for  abnormal  diagnostic  results,  and 
clinical  algorithms  for  various  conditions.  In  addition,  process  of  service  measures  such  as 
appointment  waiting  times  and  membership  application  processing  times  are  frequently  used. 
Process  measures  may  be  used  to  facilitate  comparison  of  similar  organizations  in  an  effort  to 
benchmark  performance  levels  (Siren  &  Laffel,  1996). 

Like  structure  measures,  process  measures  must  be  linked  to  clinical  outcome.  Outcomes 
measurement,  however,  has  not  always  been  clearly  defined.  Traditional  outcomes  measures 
include  infection  rates,  morbidity,  and  mortality.  The  problem  with  these  traditional  measures  is 
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that  although  outcomes  measures  are  purported  to  reflect  the  performance  of  an  entire  system, 
they  offer  little  insight  into  the  causes  of  poor  performance  (Siren  &  Laffel,  1996).  Although 
outcomes  measures  such  as  nosocomial  infection  rates,  admission  rates  for  select  diagnoses,  and 
readmission  for  the  same  condition  help  identify  potential  areas  for  improvement,  it  is  important 
to  note  that  such  measures  are  proxies  for  clinical  outcome  measurement  at  best. 

Review  of  the  literature  revealed  three  significant  trends  in  performance  measurement  as 
a  result  of  managed  care:  a  shift  from  price  to  performance  criteria,  integration  of  performance 
measurement  into  the  accreditation  process,  and  emphasis  on  long-term  prevention  and  wellness. 

Criteria  for  evaluating  managed  care  plans  have  shifted  from  price  to  performance. 

Health  plans  have  grabbed  all  of  the  “low  hanging  fruit”— the  easy  ways  to  cut  costs— like 
shorter  hospital  stays.  In  order  to  gain  new  efficiencies,  health  plans  must  begin  to  evaluate  the 
content  of  care,  not  merely  the  cost  of  care  (Ellwood  &  Lundberg,  1996).  Additionally,  health 
plans  that  historically  attracted  contracts  and  members  on  the  basis  of  cost  and  coverage  are 
being  asked  more  about  the  quality  of  services  provided.  Increasingly,  purchasers  are  just  as 
attuned  to  how  well  a  health  plan  performs  as  to  how  much  it  costs  (Marwick,  1997). 

Competition  based  on  quality  and  clinical  outcome  represents  an  attractive  way  to  be  able 
to  differentiate  between  market  players;  however,  quality  and  outcome  measurements  are  not  yet 
fully  developed  (Vladeck,  1997).  Ellwood  and  Lundberg  (1996)  stress  the  difficulty  in 
measuring  and  reporting  quality  due  to  the  lack  of  standard  measures.  Until  quality  and  outcome 
measures  can  be  reliably  quantified  and  communicated,  competition  in  these  areas  may  produce 
undesired  results. 

Probably  the  boldest  trend  in  health  plan  performance  measurement  is  the  integration  of 
performance  measurement  into  the  accreditation  process.  Both  JCAHO  and  the  National 
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Committee  for  Quality  Assurance  (NCQA)  are  striving  to  form  the  critical  link  between 
accreditation  and  outcomes  of  care  (JCAHO,  n.d./1998b;  Marwick,  1997).  Both  organizations 
envision  increased  accountability  and  credibility  in  the  accreditation  decision.  Specifically, 
JCAHO’ s  process  will: 

(1)  increase  public  and  private  sector  reliance  on  Joint  Commission  accreditation,  thereby 
building  support  for  the  consensus  development  and  use  of  standards  and  performance 
measures,  as  well  as  reducing  justification  for  duplicative  evaluation  activities; 

(2)  provide  the  Joint  Commission  with  performance  data  that  it  can  use  to  monitor 
organization  performance  on  a  continuous  basis,  help  organizations  identify  issues  that 
require  attention,  and  focus  triennial  surveys  on  areas  within  an  organization  that 
represent  the  greatest  potential  opportunities  for  improvement; 

(3)  help  identify  both  exemplary  performance  and  best  practices  to  facilitate  the 
provision  of  benchmarking  services  to  health  care  organizations;  and 

(4)  facilitate  refinement  of  the  Joint  Commission’s  performance-based  standards  and  the 
analysis  and  understanding  of  the  relationship  between  standards  and  outcomes. 

(JCAHO,  n.d./1998b,  p.  1) 

The  final  trend  in  performance  measurement  is  the  focus  on  long-term  outcome  versus 
short-term  payoff.  Bruce  Vladeck  (1997),  Administrator  for  HCFA,  emphasized  to  the  Senate 
Committee  on  Finance  that  “beyond  value  measured  on  dollars  and  cents,  managed  care  plans 
have  the  potential  to  provide  value  that  can  be  achieved  when  services  are  coordinated  and  when 
the  focus  of  care  is  on  prevention  and  wellness”  (p.  1).  To  be  effective,  health  plan  performance 
measurement  methodologies  must  take  prevention  and  wellness  into  account,  especially  with 


Medicare  populations. 
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Considering  that  nearly  60  percent  of  healthcare  dollars  are  spent  on  fewer  than  five 
percent  of  the  population  (with  over  half  of  them  being  elderly),  possible  efficiencies  gained  in 
the  areas  of  prevention  and  wellness  are  enormous  (Berk  and  Monheit,  1992;  Feldstein,  1994; 
Morgan,  Finn,  Chronister,  and  Medarametla,  1997,  p.  18).  Table  2  demonstrates  how  important 
medical  management  of  Medicare  populations  becomes.  With  inpatient  days  per  thousand  five 
times  higher,  annual  physician  contacts  two  and  a  half  times  higher,  annual  prescriptions  filled 
six  times  higher,  admissions  four  times  higher,  and  average  length  of  stays  nearly  double  than 
commercial  plans  prevention  and  wellness  deserve  great  attention  (McGraw,  1998,  p.  2-3). 

Review  of  performance  measurement  and  quality  improvement  literature  revealed  two 
previous  studies  particularly  relevant  to  the  research  question.  The  studies  focus  on  health  plan 
performance  measurement  within  two  specific  industry  segments:  a  Medicare  managed  care 
organization  (MCO)  and  an  MTF. 

Table  2 

Differences  Between  Commercial  and  Medicare  Population  Healthcare  Consumption 
Consumption  Commercial  Medicare 


Indicator _ (<65  years  of  age) _ (>65  years  of  age) 


Inpatient  Days  per  1 ,000 

200 

1000-1200 

Annual  Physician  Visits 

3.3 

8.25 

Annual  Prescriptions  Filled 

5.9 

35.4 

Admissions 

50 

200 

Average  Length  of  Stay 

3. 7-4.0 

7.0 

Note.  From  “Manage  Care  and  Reduce  Utilization  To  Succeed  in  Medicare,”  by  K.  D.  McGraw, 
1998,  The  Executive  Report  on  Managed  Care,  1 1(9),  2-4;  and  C.  S.  Sampson,  Personal 
Communication,  June  16,  1998. 
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Morgan,  Finn,  Chronister,  and  Medarametla  (1997)  collaborated  on  a  SPRY  Foundation 
examination  on  the  quality  of  care  in  HMOs  for  seniors.  The  study’s  purpose  was  to  assess  the 
current  state  of  the  art  with  regard  to  the  measurement  of  quality,  to  identify  what  had  been 
learned  from  the  measurement  efforts,  and  to  highlight  gaps  in  the  current  measurement  efforts 
(p.  1).  The  study  was  composed  of  an  in  depth  literature  review  followed  by  61  interviews. 
Individuals  interviewed  represented  consumer,  HMO,  private  foundation,  think  tank, 
professional/trade  organization,  and  purchaser/payor  perspectives. 

The  authors  contend  that  total  value  of  a  product  is  made  up  of  three  elements:  price, 
service,  and  quality.  Quality  is  by  far  the  most  complex  and  difficult  to  measure.  In  terms  of 
Donabedian’s  three  criteria  for  quality  assessment,  structure  and  process  criteria  are  readily 
measurable;  outcome  remains  elusive  (Siren  &  Laffel,  1996).  Vladeck  (1997)  echoed  their 
finding  by  stating  that  “the  science  of  outcomes  measures  is  in  its  infancy.  The  movement 
towards  better  outcomes  measures  is  critical  for  HCFA,  like-minded  purchasers,  and 
beneficiaries  in  order  to  hold  plans  and  providers  accountable  for  the  care  they  deliver”  (p.  4). 
Morgan,  Finn,  Chronister,  and  Medarametla  (1997)  conclude  that  comparable  data  do  not  exist 
to  adequately  understand  many  processes  and  outcome  measures  (p.  6). 

The  Morgan,  Finn,  Chronister,  and  Medarametla  (1997)  study,  Quality  of  Care  in  HMOs 
for  Mature  Adults,  identifies  HCFA’s  quality  of  care  measurement  instruments.  These 
instruments  comprise  one  facet  of  HCFA’s  ambitious  program  to  respond  to  public  concerns 
regarding  quality  of  care  provided  in  HMOs.  HCFA  relies  on  NCQA’s  Health  Plan  Employer 
Data  and  Information  Set,  version  3.0  (HEDIS  3.0)  for  performance  measures,  the  HEDIS  3.0 
Health  of  Seniors  questionnaire  (SF-36)  for  outcome  measures,  the  Medicare  version  of  the 
Consumer  Assessments  of  Health  Plans  Study  (CAHPS)  for  beneficiary  satisfaction,  and  a  set  of 
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clinical  outcomes  measures  developed  by  the  Foundation  for  Accountability  (FAcct).  The  study 
identified  numerous  concerns  regarding  quality  measurement  activities  for  HMOs  serving 
mature  adults: 

(a)  raising  expectations  unreasonably  high  about  the  effectiveness  of  quality 

measurement  to  improve  quality  of  care  in  the  short  term; 

(b)  ensuring  that  the  quality  measures  are  responsive  to  consumer  information  needs; 

(c)  ensuring  database  accuracy  from  which  to  develop  criteria  to  measure  quality; 

(d)  filling  in  the  gaps  in  quality  measurement;  and 

(e)  translating  the  data  into  useful  information  for  consumers,  purchasers/payers,  and 

MCOs.  (Morgan,  Finn,  Chronister,  and  Medarametla,  1997,  p.  1 1) 

Although  the  authors  admit  that  significant  progress  has  been  made  in  the  field  of  quality 
measurement,  their  key  conclusions  identify  many  shortcomings  about  quality  of  care 
measurement.  Their  findings  are:  (a)  quality  measurement  is  in  its  infancy,  (b)  there  is  a  glaring 
need  for  research  on  quality  indicators  for  seniors,  (c)  data  collection  is  a  massive  undertaking, 
(d)  enhanced  training  is  needed  at  all  levels  of  the  healthcare  industry,  and  (e)  communicating 
information  remains  a  challenge  (Morgan,  Finn,  Chronister,  and  Medarametla,  1997). 

In  contrast  to  quality  measurement  assessment  in  civilian  MCOs,  Hart  and  Connors 
(1996)  examine  the  use  of  performance  improvement  planning  within  the  context  of  an  MTF. 
Their  study  proposes  a  resourcing  decision  model  for  military  hospitals  that  incorporates  many 
of  the  same  elements  Morgan,  Finn,  Chronister,  and  Medarametla  examined. 

One  significant  difference  is  that  the  Hart  and  Connors  study  deals  with  military  budgets. 
The  MTF  commander,  equivalent  to  a  civilian  chief  executive  officer,  is  constrained  in  the 
budget  arena  by  short-term,  fixed  budgets  and  lacks  strategic  support  for  long-term  forecasting. 
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Next  year’s  MTF  budget  is  tied  directly  to  the  DHP  appropriation  approved  by  congress  and  is, 
therefore,  subject  to  significant  fluctuation.  The  concept  of  an  MTF  commander  “going  in  the 
red,”  accepting  a  short-term  loss  for  a  long-term  profit,  has  not  been  a  realistic  option.  As  the 
DoD  enters  the  competitive  arena  of  Medicare  subvention,  the  willingness  and  ability  to  take  risk 
will  become  necessaiy  (Hart  and  Connors,  1996,  p.  2). 

Information  is  central  to  the  ability  to  make  informed  business  decisions.  The 
requirement  for  performance  measurement  systems,  fed  with  valid  data,  is  a  recurring  theme — 
military  or  civilian.  “The  DoD  is  working  to  standardize  performance  indicator  development  and 
cost-analysis  systems  across  the  Army,  Navy,  and  Air  Force  so  that  relevant  cost  and  workload 
comparisons  can  be  made”  (Hart  &  Connors,  1996,  p.  2).  That  standard  is  CEIS  and  it  is  being 
deployed  across  the  DoD  as  the  official  decision  support/executive  information  support  system 
(CEIS  (EA),  1998;  Hart  &  Connors,  1996). 

Hart  and  Connors’  (1996)  conclusions  dovetail  with  those  of  Morgan,  Finn,  Chronister, 
and  Medarametla.  Successful  development  and  implementation  of  a  measurement  system 
requires  an  understanding  of  all  of  its  components  and  the  right  type  of  inputs:  cost  requires 
valid  data,  processes  must  be  well  defined,  and  outcomes  must  be  quantified  (p.  8).  Both  studies 
emphasize  the  importance  of  well-defined  performance  measurement  programs. 

Several  recurring  performance  measurement  themes  are  apparent  in  current  literature. 
Other  investigators  have  examined  performance  and  quality  measurement  opportunities  through 
related  approaches  such  as  provider  and  health  plan  report  cards,  large  administrative  databases, 
process-based  measurement,  and  clinical  and  outcome  measures.  Each  approach  shares  common 
threads  with  the  two  aforementioned  studies. 
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As  the  concept  of  performance  reporting  on  quality  has  developed,  “report  cards”  or 
“balanced  scorecards”  refer  to  standardized,  public  reports  on  health  plan  performance  (Epstein, 
1995).  While  report  cards  vary  in  format  and  content,  their  basic  purpose  is  to  provide 
customers,  purchasers  and  consumers  (all  external  to  the  healthcare  organization),  with  key, 
comparable  health  plan  information  in  an  environment  where  healthcare  organizations  are 
competing  for  market  share  (Epstein,  1995;  Health  Care  Advisory  Board  [HCAB],  1997b;  Siren 
&  Laffel,  1996;  Slovensky,  Fottler,  &  Houser,  1998).  Report  card  initiatives  are  as  varied  as  the 
health  plans  they  measure. 

Rhinehart  (in  HCAB,  1998)  maintains  that  the  health  care  industry  tracks  quality  in  three 
dimensions:  clinical  quality,  customer  satisfaction,  and  service  quality  (p.l).  Typical  report  card 
programs  include  clinical  outcomes,  cost,  functional  status,  and  satisfaction  tracking  and  analysis 
(HCAB,  1997b,  p.  2;  Slovensky,  Fottler,  &  Houser,  1998,  p.  16).  Data  elements  included  in 
report  cards  are  selected  according  to  the  anticipated  audience  (Slovensky,  Fottler,  &  Houser, 
1998).  For  example,  many  hospital-based  report  cards  intended  for  the  community  further  divide 
quality  indicators  to  measure  access  to  care,  clinical  outcomes,  community  needs  fulfillment, 
financial  indicators,  number  of  patients  served,  and  patient  satisfaction.  The  clinical  data 
included  in  these  reports  are  intended  to  measure  the  quality  of  care;  however,  these  measures 
tend  to  focus  on  health  plan  administrative  processes  and  preventative  treatments  such  as 
mammographic  screening  and  immunization  rates  rather  than  on  outcomes  data  (Epstein,  1995; 
HCAB,  1998). 

Slovensky,  Fottler,  and  Houser  (1998)  recently  reviewed  report  card  development  and 
implementation.  Their  research  places  report  card  initiatives  into  five  categories:  (a)  those 
mandated  by  state  agencies,  (b)  those  developed  by  health  plans,  (c)  those  requested  by 
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purchaser  coalitions,  (d)  those  resulting  from  collaborative  efforts  of  not-for-profit  organizations, 
and  (e)  those  produced  by  voluntary  provider  alliances.  Prime  examples  of  report  card  initiatives 
are  DoD’s  Quality  Management  Report  (QMR)  and  NCQA’s  Quality  Compass  program. 

The  QMR  was  one  of  the  first  formalized  MHS  system-wide  performance  reports.  The 
report  provided  a  forum  in  which  to  highlight  significant  quality  improvement  activities  and 
accreditation  efforts.  A  “dashboard”  of  quality  performance  indicators  addressing  hospital 
accreditation  status,  practitioner  licensure  status,  physician  board  certification,  beneficiary 
satisfaction,  inpatient  utilization  rates,  and  preventable  admissions  [more  accurately  indicators  of 
preventable  admissions  such  as  asthma  hospitalization  rates]  formed  the  heart  of  the  report 
(ODSD  (HA),  1997a,  1997b). 

Quality  Compass  collects  information  on  100  performance  measures,  many  of  which 
come  from  HEDIS.  HEDIS  collects  information  on  75  standardized  performance  measures  to 
compare  over  300  managed  health  care  plans  (NCQA,  1997b,  1997c;  Spoeri  &  Ullman,  1997).  A 
second,  closely  related  example  is  the  JCAHO  sponsored  Quality  Check  program.  JCAHO 
(1997)  promises  that  “Quality  Check  will  provide  you  with  information  that  lets  you  know  where 
your  health  care  organization  stands  when  it  comes  to  the  quality  of  care  it  provides”  (p.  1). 
Primarily,  both  programs  (as  well  as  HEDIS)  examine  administrative  processes  as  proxies  for 
clinical  outcome — measures  useful  to  external  customers  versus  health  plan  managers. 

As  seen  in  the  NCQA  example,  many  report  cards  are  derived  from  administrative  data, 
data  resulting  from  the  administration  of  healthcare  delivery — enrolling  members  into  insurance 
plans  and  processing  claims  for  services  provided  (Iezzoni,  1997).  Medicare’s  large 
administrative  databases,  coupled  with  emerging  computer  technologies,  facilitated  early 
performance  measurement  in  the  1960’s  and  1970’s.  For  the  first  time,  investigators  had  access 
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to  significant  sets  of  data  and  were  able  to  employ  powerful  epidemiological  methods  in  their 
analyses  of  healthcare  delivery  (Siren  &  Laffel,  1996).  Although  administrative  data  were  never 
intended  for  use  in  quality  assessment,  their  usefulness  as  screening  tools  is  apparent  in  both 
process  and  structure  measures  (Iezzoni,  1997).  Outcome  measures  with  administrative  data  are, 
however,  another  story. 

Administrative  data  offer  investigators  several  attractive  features:  they  are  readily 
available,  inexpensive  to  acquire,  computer  readable,  and  easily  compared  across  populations. 
Outcomes  measurement,  however,  requires  detailed  clinical  information  that  is  generally  buried 
deep  within  paper  medical  records.  The  costs  of  acquiring  and  maintaining  clinical  data  is, 
therefore,  significantly  higher  (Iezzoni,  1997).  “Widespread  quality  assessment  [outcomes 
measurement]  demands  a  tradeoff:  the  credibility  of  clinical  data  versus  the  expense  and 
feasibility  of  data  collection”  (Iezzoni,  1997,  p.  2). 

Since  detailed  clinical  information  is  typically  not  found  in  large,  automated  databases, 
careful  use  of  existing  process  measures  along  with  developing  electronic  medical  records  may 
serve  as  a  bridge  between  current  measurement  methodologies  and  the  “utopia”  of  clinical 
outcomes  measurement  (Nerenz,  1997).  Palmer  (1997)  offers  such  an  alternative  to  pure 
outcomes  measurement — process-based  measures  of  quality  (p.  1).  Unlike  report  card  initiatives, 
process-based  outcome  measurements  provide  health  plan  executives  with  critical  information 
needed  for  plan  management. 

Process-based  measures  attempt  to  quantify  how  well  specific  processes  along  the  care 
continuum  are  executed.  The  processes  measured  are  those  that  experts  agree  can  be 
scientifically  shown  to  improve  clinical  outcome.  The  level  of  scientific  evidence  required  for 
publicly  released  measures  of  quality  should  be  particularly  high.  Process-based  measures  of 
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health  care  quality  can  be  constructed  by  combining  evidence  from  multiple  studies  in  order  to 
compile  evidence-based  guidelines  on  clinical  practices  (Epstein,  1995;  Palmer,  1997,  p.2). 

These  clinical  practices  outline  analytically  the  specific  procedures  to  be  followed  for 
suspected  diagnoses  or  primary  complaints.  Historically,  such  guidelines,  or  clinical  protocols, 
have  been  used  as  the  basis  for  modeling  and  computerized  teaching  (Williams  &  Torrens, 

1993).  In  addition  to  the  creation  of  guidelines,  criteria  for  quality  are  also  formulated  to  identify 
candidate  patients.  Conceptually,  the  number  of  patients  identified  to  receive  the  guideline- 
related  care  is  divided  by  the  total  number  of  patients  who  were  eligible  to  receive  the  care  (or 
those  who  should  have  received  the  care).  Although  moving  such  measurement  initiatives  from 
theoretical  to  operational  environments  is  confounded  by  information  systems’  inability  to 
collect  (or  project)  the  denominator,  the  basic  construct  can  lead  managers  closer  toward 
outcomes  measurement.  The  concept  behind  guideline-related  care  is  intended  to  result  in  rates 
that  may  be  used  to  describe  how  well  the  healthcare  system  “worked”  in  relation  to  how  well  it 
should  have  worked  (Palmer,  1997). 

Obviously,  no  system  can  be  expected  to  correctly  predict  how  each  case  should  be 
treated.  Sources  of  variation  include  patient  and  provider  preference,  compliance  with  provider 
recommendations,  and  patients  don’t  always  fit  neatly  into  guidelines.  Process-based 
measurement  also  suffers  from  the  same  handicap  that  clinical  outcomes  measurement  does:  the 
difficulty  in  collecting  detailed  clinical  data  (Palmer,  1997). 

Fortunately,  large-scale  process-based  measurement  is  becoming  more  feasible. 
Advances  in  technology  are  allowing  easier  integration  of  clinical  medical  information  systems. 
Merging  inpatient  and  outpatient  information  with  laboratory  and  pharmacy  databases  is  an 
important  step  in  ultimately  linking  all  patient  information  (Palmer,  1997). 
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The  majority  of  the  literature  reviewed  concentrated  on  commercial  health  plan 
performance  measurement.  Little  work  has  been  published  specifically  on  performance 
measurement  of  Medicare  health  plans,  plans  that  serve  an  especially  vulnerable  population. 
Lavizzo-Mourey  and  Mackenzie  (1996)  explore  managed  care’s  potential  effects  on  minority 
populations  and  postulate  that,  while  managing  healthcare  utilization  and  narrowing  consumer 
choice  can  be  effective  cost-saving  strategies,  they  also  present  a  potential  threat  to  quality  of 
care  for  these  groups  in  particular.  They  approach  the  subject  of  performance  measurement  from 
a  slightly  different  view — ’’cultural  competence.” 

As  we  approach  the  year  2000,  roughly  25  percent  of  the  U.S.  population  will  be 
members  of  “minority”  groups.  Since  minority  groups  are  not  monolithic,  the  considerable 
diversity  that  exists  within  U.S.  subpopulations  must  be  factored  into  the  guidelines  of  quality. 
On  this  basis,  Lavizzo-Mourey  and  Mackenzie  (1996)  conceptualize  cultural  competence  as  the 
awareness  and  integration  of  three  population-specific  issues:  health-related  beliefs  and  cultural 
values,  disease  incidence  and  prevalence,  and  treatment  efficacy  (p.  1). 

Daniel  (1996)  may  have  unknowingly  touched  on  this  important  issue  when  he  observed 
that  “given  the  right  demographics,  location,  utilization,  and  cost  efficiency,  hospitals]  and 
health  systems  will  recognize  financial  and  operational  gains  by  transitioning  themselves  from 
the  passive  recipient  to  the  active  administrator  of  the  Medicare  dollar”  (p.  407).  Managed  care 
performance  measurement  must  be  flexible  enough  to  account  for  not  only  demographic  but  also 
cultural  difference,  especially  when  entering  new  markets  with  new  products. 

As  physicians  establish  guidelines  for  quality  of  care  that  support  the  changing  managed 
care  environment,  they  must  address  two  important  trends.  First,  managed  care  was  formerly 
confined  to  a  predominately  middle  class  population  whereas  it  now  extends  to  diverse  and  more 
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vulnerable  groups  including  Medicare  and  Medicaid.  Additionally,  a  managed  care  environment 
focuses  on  covered  lives — the  individual  patient  must  not  be  lost  in  the  shuffle  (Lavizzo-Mourey 
and  Mackenzie,  1997,  p.  1). 

Like  Lavizzo-Mourey  and  Mackenzie,  McHomey  (1997)  illustrates  a  potential 
performance  measurement  pitfall.  McHomey’s  basic  question  is  why  use  generic  surveys  with 
“non-generic”  populations? 

Researchers  have  designed  numerous  tools  that  rely  on  self-reporting  to  measure 
everything  from  functional  status  to  customer  satisfaction.  Managed  care  movements,  such  as 
outcomes  measurement,  have  dramatically  increased  the  use  of  generic  surveys.  McHomey 
(1997)  proposes  the  use  of  emergent  technology  to  improve  the  feasibility  and  utility  of 
including  patient-centered  data  in  large  administrative  databases  (p.  1). 

Generic  health  status  assessments  were  developed  in  the  group-testing  tradition.  Features 
of  such  tools  include  a  fixed  set  of  items  for  all  respondents,  items  written  to  represent  the 
“norm,”  and  as  few  items  as  possible.  In  addition,  acceptable  standards  of  validity  and  reliability 
of  tools  with  few  items  is  best  achieved  by  selecting  items  that  are  fairly  homogeneous.  Two 
consequences  result:  fixed-length  surveys  either  bore  healthy  respondents  or  frustrate  more 
impaired  respondents.  Since  item  selection  is  geared  toward  middle-of-the-road,  the  end  points 
of  the  health  continuum  are  poorly  defined  with  such  instruments  (Cooper  &  Emory,  1995; 
McHomey,  1997,  p.  4).  McHomey’s  answer  to  this  phenomenon  is  equiprecise  measurement  for 
generic  health  concepts. 

Equiprecise  tests  yield  measures  of  equal  precision  at  all  levels — equal  precision  for  the 
sickest  and  healthiest.  This  type  of  measurement  is  possible  through  the  conjoint  use  of 
computerized  adaptive  testing  and  the  measurement  theory  known  as  “item  response  theory” 
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(McHomey,  1997,  p.  5).  Patient  survey  information,  to  include  elusive  elements  such  as 
behavioral  and  emotional  function  data,  can  be  tailored  to  the  individual  and  can  be  collected  for 
inclusion  into  outcomes  databases,  often  in  half  the  time  as  standard  pencil-and-paper  surveys. 
The  power  of  computerized  adaptive  testing  has  been  demonstrated  with  knowledge-based 
testing,  such  as  the  Graduate  Management  Admission  Test.  Testing  computer  software  selects 
future  questions  based  on  answers  already  provided.  Items  that  either  do  not  apply  or  have 
already  been  covered  adequately  are  not  offered  to  the  person  being  surveyed.  Therefore,  such 
tests  can  reduce  testing  time  by  50  to  75  percent  and  are  capable  of  providing  rapid  scoring  and 
feedback  (McHomey,  1997).  The  inclusion  of  equiprecise  measurement  of  health  concepts  in  a 
Medicare  managed  care  setting  has  the  potential  to  make  significant  contribution. 

The  promise  computerized  equiprecise  measurement  offers  is  also  tempered  by  the  fact 
that  the  measurement  is  computer-based.  The  issues  of  computer  equipment  availability  and 
many  seniors’  unfamiliarity  with  computer  technology  must  be  addressed  to  prevent  failure. 
Although  adaptation  of  equiprecise  measurement  to  a  paper-based  format  looses  the  efficiency 
of  automation,  the  near-term  utility  of  such  a  tool  is  an  attractive  alternative  to  traditional 
surveys. 

Eight  key  measurement  projects  relevant  to  the  research  question  were  identified:  four 
civilian  performance  measurement  projects,  two  performance  indicator  research  studies,  and  two 
MHS  performance  measurement  programs. 

The  “flag  ship”  performance  measurement  initiative,  the  Quality  Indicator  (QI)  Project 
(more  commonly  known  as  “Maryland  Indicators”),  got  its  start  in  1985  as  a  pilot  project  of 
seven  Maryland  hospitals.  The  ultimate  goal  of  the  QI  Project  was  to  serve  as  a  tool  for 
overseeing  the  quality  of  care  and  for  identifying  opportunities  for  improvement.  The  clinical 
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orientation  of  the  QI  Project  is  apparent  in  the  15  inpatient  and  ambulatory  outcomes-based 
indicators  included  in  the  primary  data  set.  Additional  indicators  being  examined  include 
psychiatric,  pediatric,  long-term  care,  and  process  indicators  (Maryland  Hospital  Association 
(MHA),  1998a). 

Although  the  QI  Project  is  not  “intended  to  be  used  to  establish  performance  thresholds 
or  standards  of  care”  (MHA,  1998a,  p.  1.),  its  impact  on  subsequent  performance  measurement 
initiatives  can  be  seen  in  one  of  the  most  current  programs:  Oryx.  Oryx  is  JCAHO’s  latest  plan 
to  base  healthcare  organization  accreditation,  in  part,  on  provider  performance.  Eleven  of  the 
original  15  QI  Project  indicators,  as  well  as  several  of  the  psychiatric  and  long-term  care 
indicators,  are  mirrored  on  the  Oryx  approved  measures  list  (HCAB,  1997c;  MHA,  1998b;  and 
Moore,  1997). 

One  of  the  first  comprehensive  performance  measurement  initiatives  to  illustrate  the  need 
to  objectively  gauge  health  delivery  system  performance  was  the  Consortium  for  Research  on 
Indicators  of  System  Performance  (CRISP).  Henry  Ford  Health  System’s  Chief  Executive 
Officer,  Gail  Warden,  launched  the  CRISP  project  in  1991,  by  posing  the  question:  “How  do  we 
know  if  our  system  is  doing  a  good  job”  (Nerenz,  Zajac,  and  Rosman,  1993,  p.  577)? 

The  recognition  that  large,  integrated  health  systems  were  becoming  increasingly 
numerous,  and  that  such  systems  needed  to  be  able  to  measure  overall  performance  in  support  of 
public  accountability,  resulted  in  an  initial  conceptual  framework  of  91  quantitative  performance 
indicators  (Nerenz,  Zajac,  and  Rosman,  1993).  Although  CRISP  is  no  longer  an  active  research 
project  (as  of  May  1998),  the  Consortium  served  as  the  catalyst  for  dozens  of  initiatives  to 
follow.  Its  current  direction  has  shifted  from  inter-health  system  measurement  and  comparison  to 
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intra-health  system  outcomes  measurement  research  (L.  Pietrantoni,  personal  communication, 
May  22,  1998). 

Although  touted  as  the  most  well  known  and  most  widely  used  health  plan  performance 
measurement  tool  in  the  industry,  HEDIS  actually  serves  purchasers  of  health  care  services  more 
than  the  managers  of  those  services.  The  current  version,  HEDIS  3.0,  was  designed  to  help 
consumers  evaluate  the  quality  of  managed  care  plans  by  providing  standardized  performance 
measures  focusing  on  outcomes.  HEDIS  compares  the  performance  of  more  than  330  MCOs 
across  the  nation  in  eight  areas  of  performance  ranging  from  access  and  quality  of  care  to  health 
plan  stability  (NCQA,  1997a,  1997c;  Werner,  1996). 

While  designed  for  the  healthcare  consumer,  HEDIS  is  widely  used  by  MCOs  because  of 
the  measures’  “track  history...  and  statistical  properties”  (NCQA,  1997a,  p.  2)  needed  to 
benchmark  performance.  HEDIS  has  established  itself  in  the  industry  and  will  continue  to  serve 
as  a  foundation  for  future  performance  measurement  initiatives.  The  next  version  of  HEDIS, 
currently  referred  to  as  HEDIS  1999,  is  due  to  be  released  in  the  summer  of  1998.  The  proposed 
changes  fill  gaps  previously  identified  in  version  3.0  (NCQA,  1998). 

The  only  proprietary  performance  measurement  system  included  in  this  study  is  the 
Comprehensive  Healthcare  Analysis  and  Management  Program  (CHAMP)  from  William  M. 
Mercer,  Incorporated.  One  of  countless  commercially  available  systems,  CHAMP  represents  the 
operational  end  of  the  performance  measurement  system  continuum.  CHAMP,  designed 
primarily  as  a  health  plan  financial  decision  support  system,  provides  performance  indicators  at 
the  health  plan  operations  level.  The  program  supports  plan  management  in  the  areas  of  benefit 
design,  utilization  management,  and  provider  price  contracts  (Mercer,  1997). 
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Two  HCAB  custom  research  projects,  Internal  Performance  Indicators  and  Hospital- 
Wide  Quality  Data  Indicators  provide  insight  to  the  content  and  organization  of  hospital-based 
performance  indicators.  Although  specific  indicator  specifications  are  not  provided,  the  studies 
illustrate  how  performance  indicators  generally  fall  within  the  broad  categories  of  community 
service,  financial  performance,  quality  statistics,  and  general  customer-related  indicators 
(HCAB,  1995, 1996a). 

The  two  military-specific  performance  measurement  initiatives  identified  represent  the 
state-of-the  art  for  the  MHS:  the  MHS  Performance  Report  Card  (MHS  PRC),  and  the  Air  Force 
Medical  Service  (AFMS)  Performance  Measurement  Tool  (PMT). 

In  late  1995,  senior  staff  members  of  OSD  (HA)  formulated  an  initial  series  of 
performance  measures  in  support  of  the  MHS  strategic  plan.  The  purpose  of  these  metrics  was  to 
provide  healthcare  managers  at  the  DoD  corporate  level  with  a  measurement  tool  that  would 
help  them  evaluate  the  effectiveness  of  the  MHS  as  the  DoD  migrated  into  a  managed  care 
environment.  The  MHS  PRC  was  later  developed  to  allow  the  resulting  measures  to  be  examined 
at  the  MTF  level  by  all  managers,  particularly  those  at  the  point  of  healthcare  delivery  (Health 
Services  Analysis  and  Measurement  (HSAM),  1997). 

The  AFMS  PMT  was  developed  in  response  to  the  recognized  lack  of  any  standardized 
performance  measurements  and  the  incentives  to  achieve  them  within  the  AFMS  (AFMS,  1998). 
The  Air  Force  Medical  Systems  roundtable  concluded  in  early  1997,  “these  deficiencies  create 
significant  barriers  to  improving  AFMS  quality,  service,  and  cost  efficiency”  (AFMS,  1998). 
Further,  “the  creation  of  an  AFMS  PMT  that  uses  standardized  metrics  and  automates  data 
collection  (particularly  for  the  AFMS  as  an  accountable  health  care  plan  with  an  enrolled 
population)  would  be  a  crucial  first  step  in  meeting  [AFMS]  goals”  (AFMS,  1998). 
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Although  each  performance  measurement  initiative  evaluated  was  designed  for  distinct 
purposes,  all  share  the  same  overall  goal:  to  link  performance  measurements  with  organizational 
missions  and  visions. 


Method  and  Procedures 

To  adequately  address  the  research  question,  “are  current  MHS  performance 
measurement  systems  adequate  to  evaluate  Medicare  Subvention  Demonstration  success?”  the 
following  objectives  were  explored: 

(a)  What  data  elements  are  readily  available  in  MHS  systems  and  how  do  they  compare 

with  industry  standards? 

(b)  What  data  set  is  required  for  adequate  performance  measurement  and  comparison? 

(c)  Do  current  MHS  performance  measurement  systems  capture  the  data  required  to 

measure  and  relate  health  plan  performance? 

This  is  a  Level  1,  descriptive  study.  A  qualitative  methodology — a  detailed  investigation 
of  existing  performance  measurement  systems — was  determined  to  be  the  most  appropriate 
approach  because  the  lack  of  MHS  Medicare  HMO  experience  precluded  the  formulation  of  a 
theoretically  derived  hypothesis.  The  sampling  strategy  chosen  includes  elements  of  both 
convenience  (unrestricted,  nonprobability)  and  snowball  (restricted,  nonprobability)  techniques 
due  to  the  largely  exploratory  nature  of  the  research  question. 

Existing  performance  measurement  schemes  found  within  civilian  and  military 
healthcare  organizations  form  the  initial  sampling  frame.  Initial  study  cases  were  identified 
through  two  convenience  methods:  literature  review  via  multiple  Internet  search  engines,  and 
interviews  with  healthcare  executives  in  the  St.  Louis,  MO  area.  The  following  Internet  search 
engines  and  World-Wide-Web  (WWW)  sites  were  used: 
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(a)  HealthGate:  http://www.healthgate.com/ 

(b)  MedConnect:  http://www.infotrieve.com/medconnect/ 

(c)  Dogpile:  http://www.dogpile.com/ 

(d)  The  Advisory  Board  Company:  http://www.advisory-hcab.com/cgi-bin/hcab.cmd/ 

(e)  DoD  Health  Affairs:  http://www.osd.ha.mil/ 

(f)  ACP  Online:  http://www.acponline.org/ 

(g)  NCQA:  http://www.ncqa.org/ 

(h)  JCAHO:  http://www.jcaho.org/ 

(i)  HCFA:  http://www.hcfa.gov/ 

Internet  search  engines  and  WWW  site  search  utilities  were  used  to  retrieve  documents 
and  world-wide-web  sites  that  responded  to  the  following  target  terms: 

(a)  health  plan  performance 

(b)  health  plan 

(c)  performance  measurement 

(d)  performance  indicators 

(e)  quality  management 

(f)  continuous  quality  improvement 

(g)  quality  indicators 

(h)  report  cards 

(i)  balanced  scorecards 

(j)  HEDIS 

Final  study  cases  were  identified  through  snowball  sampling  (Cooper  &  Emory,  1995, 
p.  230)  of  the  initial  group.  For  journals  and  WWW  documents,  a  reverse  search  for 
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bibliographic  sources  was  conducted  to  identify  additional  study  cases.  Journal  selection 
criterion  was  restricted  to  the  initial  search  topic  list  and  dated  December  1993  or  later. 

In  addition  to  an  extensive  literature  review,  a  series  of  interviews  were  conducted.  The 
majority  of  the  interviewees  were  selected  from  the  St  Louis  market  because  of  easy  access, 
convenient  location,  and  diverse  perspectives  (See  Table  3).  Data  elements  (individual 
performance  indicators)  extracted  from  the  various  health  plan  performance  measurement 
schemes  comprise  the  final  study  sample. 

MHS  and  civilian  performance  measurement  systems  were  compared  and  contrasted  in 
an  effort  to  determine  the  optimal  set  of  data  elements  required  for  Medicare  health  plan 
performance  measurement.  Data  elements  were  compiled  from  the  following  sources: 

(a)  primary  and  secondary  literature  reviews;  (b)  interviews  with  civilian  health  plan  executives 
(Renaissance  Health  Plan,  Partners  Health  Plan  of  the  Midwest,  Care  Partners  Health  Plan,  and 
Health  Management  Partners);  and  (c)  site  visits  at  Scott  AFB,  IL  and  Keesler  AFB,  MS. 
Performance  indicators  present  in  MHS  systems  were  identified  and  compared  to  those  elements 
representing  industry  norms  from  civilian  systems. 

Primary  initiatives  compared  include:  (a)  the  Air  Force  Medical  Service  (AFMS) 
Performance  Measurement  Tool  (PMT),  (b)  the  MHS  Performance  Report  Card  (PRC),  (c)  the 
Consortium  Research  on  Indicators  of  System  Performance  (CRISP),  (d)  the  Comprehensive 
Healthcare  Analysis  and  Management  Program  (CHAMP),  (e)  the  Health  Plan  Employer  Data 
and  Information  Set  (HEDIS),  and  (f)  two  related  research  projects  from  the  Health  Care 
Advisory  Board  (HCAB).  Performance  indicator  specifications  for  each  model  are  at  Appendices 
C  through  I  respectively. 
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Table  3 


Individuals  and  Organizations  Interviewed 
Name 


Title(s) 


Lt  Col  Tom  Ardoline 
Dr.  Joseph  Avelone 
Mr.  Francis  Brissette 
Mr.  Emmette  Craft 
Mrs.  Sandy  Graff 


Commander,  TRICARE  Flight,  Keesler  Medical  Center,  Keesler  AFB,  MS 
President,  Renaissance  Health  Plans,  MA 
VP  Finance,  Renaissance  Health  Plans,  MA 

Director,  Managed  Care  Contracting,  BJC  Health  System,  St  Louis,  MO 

Director,  Medical  Management,  Health  Management  Partners.  LLC,  St  Louis, 
MO 


Lt  Col  Bemie  Ken- 
Dr.  John  Lynch 
Mr.  Dennis  Mathis 
Maj  Tim  McCormick 
Ms.  Lisa  Pietrantoni 
Mr.  Charles  Sampson 

Dr.  Deb  Zimmerman 


Director,  Technology  Insertion  &  Benefits  Realization,  Scott  AFB,  IL 
Medical  Director,  Health  Management  Partners,  LLC,  St  Louis,  MO 
President,  Health  Partners  of  the  Midwest 

Commander,  TRICARE  Flight,  375th  Medical  Group,  Scott  AFB,  IL 

Center  for  Health  System  Studies,  Henry  Ford  Health  System 

VP,  Managed  Care,  BJC  Health  System;  President,  Care  Partners  HMO;  and 
President,  BJC  Dental  Plan,  St  Louis,  MO 

Medical  Director,  Health  Partners  of  the  Midwest,  St  Louis,  MO _ 


A  Microsoft  Access  database  consisting  of  individual  performance  indicators  from  each 
of  the  initiatives  was  compiled.  Each  indicator-initiative  combination,  representing  a  unique 
database  record,  was  subjectively  coded  classifying  it  as  either  a  military  or  civilian  initiative 
and  as  being  a  structure,  process,  or  outcome  criteria  measure.  Indicators  were  also  coded  as  one 
of  12  general  measurement  categories:  financial  performance,  customer  satisfaction,  quality  (to 
include  preventive,  clinical  outcome,  disease  management,  and  access  to  care),  community 
service,  utilization,  enrollment,  population,  and  other. 
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A  recognized  limitation  in  the  coding  convention  is  that  indicators  were  coded  as  falling 
exclusively  into  one  measurement  criteria  and  one  performance  indicator  category:  few 
indicators  belong  in  only  one  category. 

A  variety  of  Microsoft  Excel  logical  and  reference  table  commands  were  used  to  convert 
file  entries  to  a  Statistical  Package  for  Social  Sciences  (SPSS),  version  7.0,  compatible  format. 
Reliability  of  data  transfer  was  confirmed  by  comparing  categorical  frequencies  from  the 
original  Access  file  and  the  resulting  SPSS  file — resulting  in  a  100  percent  accuracy  of  data 
transfer.  Data  labels,  values,  and  codes  may  be  found  in  the  Data  Codebook  at  Appendix  J. 

The  evaluation  scheme  for  the  study  is  comprised  of  three  steps  directly  related  to  the 
three  objectives.  The  first  evaluation  point  (a)  determines  what  data  elements  are  readily 
available  in  MHS  systems  and  how  they  compare  with  industry  standards.  Objective  and 
evaluation  point  (b)  results  in  a  model  health  plan  performance  measurement  system  as  a  result 
of  the  “best  practice”  review  and  an  informal  consensus  of  interviewed  healthcare  executives. 
Finally,  (c)  the  degree  which  MHS  performance  measurement  systems  compare  to  the  proposed 
“best  case”  model  will  be  reported. 

Reliability  of  the  study  is  demonstrated  by  the  extent  of  agreement  between  the  various 
literature  and  interview  sources  used  to  compile  the  best  case  model.  Agreement  on  critical 
performance  measurement  data  elements  and  performance  indicators  among  a  variety  of  case 
studies  and  health  plans  build  support  for  acceptable  reliability  of  the  proposed  model.  Validity, 
however,  will  be  more  difficult  to  substantiate. 

Since  the  model  proposed  will  be  developed  from  multiple,  varied  input  from  previous 
research,  case  studies,  and  operational  health  plans,  external  validity  will  be  high.  Study 
recommendations  may  be  successfully  generalized  within  military  or  civilian  healthcare 
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organizations  seeking  to  monitor  Medicare  health  plan  performance.  Extracting  enough  detail 
from  successful  health  plans  and  performance  measurement  systems  to  form  a  complete  and 
representative  collection  of  data  elements  governs  content  validity.  Content  validity  will  be 
dependent  largely  on  the  completeness  of  the  literature  review;  however,  it  is  important  to  note 
that  the  exploratory  nature  of  the  study  does  not  represent  an  exhaustive  review.  Consensus 
among  interview  sources  that  the  data  set  compiled  is  complete  supports  overall  study  validity. 

Results 

The  study  sample  consists  of  318  records  (N=318)  compiled  from  the  examination  of  six 
performance  measurement  initiatives:  two  MHS  performance  measurement  programs,  three 
civilian  performance  measurement  projects,  and  two  performance  indicator  research  studies 
(treated  as  one  initiative).  Although  each  record  is  a  unique  combination  of  performance 
initiative  and  indicator,  duplication  of  indicator  function  between  initiatives  reduces  the  number 
of  unique  performance  indicators  to  approximately  180.  Furthermore,  as  demonstrated  in 
Table  4,  individual  performance  indicators  can  be  subjectively  grouped  into  28  general  areas 
based  on  indicator  descriptions  from  each  initiative  investigated.  The  most  common  indicator 
categories  are  Financial  Performance  (6  of  7  measurement  schemes),  Customer  Satisfaction  (5  of 
7),  Quality  of  Care  (4  of  7),  Community  Service  (3  of  7),  and  Utilization  (3  of  7). 

Within  the  318  indicators,  83.3  percent  represent  process  measures,  1 1.9  percent 
represent  outcome  measures,  and  the  remaining  4.7  percent  are  structure  measures.  The 
distribution  of  indicators  employed  by  each  performance  measurement  scheme  is  provided  in 
Table  5  by  indicator  criteria  and  by  indicator  category. 
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Table  4 

Matrix  of  General  Areas  Covered  by  Select  Performance  Measurement  Schemes 


CRISP 


Provider  Analysis 

Prevention 

Indicators 

Quality  of 
Care 
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The  data  elements  that  are  readily  available  in  MHS  systems  are  detailed  in  Appendix  C 
and  D.  Data  elements  representing  civilian  industry  standards  are  detailed  in  Appendices  E 
through  I.  Comparison  of  individual  elements  (indicators)  proved  problematic  due  to  wide 
variability  in  indicator  construction;  however,  comparisons  of  the  indicators  by  criteria  and  by 
category  are  provided  (Table  5).  Comparison  of  the  total  group,  military  only,  civilian  only,  and 
civilian  minus  HCAB  indicator  subsets  at  Table  6  demonstrate  general  agreement  in  the 
composition  of  military  and  civilian  performance  measurement  systems  in  terms  of  the  general 
areas  measured. 

The  fourth  indicator  set  in  Table  6,  civilian  minus  HCAB,  is  provided  as  an  alternate 
comparison  group.  Unlike  the  five  other  performance  measurement  initiatives,  the  HCAB  studies 
represent  secondary  research  and  lack  original,  detailed  performance  indicator  specifications. 
Additionally,  the  HCAB  studies  concentrate  on  inpatient  hospital  rather  than  health  plan 
performance  assessment.  Although  useful  in  determining  the  framework  of  general  performance 
measurement  systems,  exclusion  of  these  indicators  provides  a  closer  comparison  of  operational 
health  plan  performance  measurement  systems. 

The  relative  small  sample  size  (n<500)  made  one-sample  hypothesis  testing  of 
percentages  impractical  (Sanders,  1995,  p.  307);  however,  cursory  examination  of  the  results 
(Table  6)  illustrates  agreement  between  the  military  and  civilian  measurement  schemes  in  terms 
of  the  general  organization  of  measurement  areas.  Each  measurement  scheme  uses  a  variety  of 
performance  indicators  within  similar  evaluation  areas  to  assess  organizational  performance. 
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Table  5 

Distribution  of  Indicators  Employed  by  Select  Performance  Measurement  Schemes  (N=318) 


CIVILIAN  _  MILITARY 


INDICATOR 

CHARACTERISTIC 

CRISP 

% 

(n=18) 

HCAB 

% 

(n=165) 

CHAMP 

% 

(n=24) 

HEDIS 

% 

(n=60) 

AFMS  PMT 
% 

(n=27) 

MHS  PRC 
% 

(n=24) 

INDICATOR  CRITERIA 

Process 

77.8 

80.0 

91.7 

90.0 

85.2 

83.3 

Outcome 

22.2 

17.6 

8.3 

5.0 

0.0 

0.0 

Structure 

0.0 

2.4 

0.0 

5.0 

14.8 

16.7 

INDICATOR  CATEGORY 

Financial  Performance 

16.7 

29.7 

54.2 

5.0 

14.8 

0.0 

Customer  Satisfaction 

5.6 

9.7 

0.0 

3.3 

25.9 

12.5 

Quality 

55.6 

21.7 

25.0 

46.7 

44.4 

62.5 

Quality — General 

16.7 

3.0 

4.2 

6.7 

3.7 

12.5 

Quality — Preventive 

11.1 

4.8 

8.3 

16.7 

33.3 

50.0 

Quality — Outcome 

27.8 

13.9 

8.3 

3.3 

0.0 

0.0 

Quality — Disease  Mgt 

0.0 

0.0 

4.2 

8.3 

7.4 

0.0 

Quality — Access  to  Care 

0.0 

0.0 

0.0 

11.7 

0.0 

0.0 

Community  Service 

5.6 

2.4 

0.0 

0.0 

0.0 

0.0 

Utilization 

16.7 

19.4 

8.3 

28.3 

0.0 

16.7 

Enrollment 

0.0 

3.6 

8.3 

6.7 

7.4 

4.2 

Population 

0.0 

3.6 

4.2 

5.0 

3.7 

0.0 

Other 

0.0 

9.7 

0.0 

5.0 

3.7 

4.2 
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Table  6 


Summary  of  Health  Plan  Performance  Measurement  Data  Elements  from  MHS  and  Civilian 


Measurement  Schemes 

INDICATOR 

CHARACTERISTIC 

Total 

Initiatives 

% 

(N=318) 

Military 

Initiatives 

% 

(n=51) 

Civilian 

Initiatives 

% 

(n=267) 

Civilian-HCAB 

Initiatives' 

% 

(n=102) _ 

INDICATOR  CRITERIA 

Process 

83.3 

84.3 

83.1 

88.2 

Outcome 

11.9 

0.0 

14.2 

8.8 

Structure 

4.7 

15.7 

2.6 

2.9 

INDICATOR  CATEGORY 

Financial  Performance 

22.6 

7.8 

25.5 

18.6 

Customer  Satisfaction 

9.1 

19.6 

7.1 

2.9 

Quality 

33.1 

52.9 

29.9 

43.1 

Quality — General 

5.3 

7.8 

4.9 

7.8 

Quality — Preventive 

13.5 

41.2 

8.2 

13.7 

Quality — Outcome 

10.1 

0.0 

12.0 

8.8 

Quality — Disease  Mgt 

2.5 

3.9 

2.2 

5.9 

Quality — Access  to  Care 

2.2 

0.0 

2.6 

6.9 

Community  Service 

1.6 

0.0 

1.9 

1.0 

Utilization 

18.2 

7.8 

20.2 

21.6 

Enrollment 

4.7 

5.9 

4.5 

5.9 

Population 

3.5 

2.0 

3.7 

3.9 

Other 

6.6 

3.9 

7.1 

2.9 

1  Note.  Frequencies  are  provided  for  civilian  initiatives  without  the  Health  Care  Advisory 
Board  figures  for  additional  comparison  due  to  the  lack  of  indicator  specifications  in  the  studies. 
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At  first  glance,  significant  differences  exist  in  the  weight  applied  to  measurement 
categories  as  well  as  the  distribution  of  structure,  process,  and  outcomes  criteria  within  those 
categories.  However,  categorization  of  individual  performance  indicators,  as  described  in 
methods,  contributes  to  much  of  the  apparent  difference  between  measurement  schemes. 
Although  indicators  were  exclusively  placed  in  one  of  twelve  categories,  many  indicators  could 
easily  have  fit  in  multiple  categories.  Additionally,  dividing  lines  between  structure,  process,  and 
outcomes  measures  are  not  well  defined. 

Apparent  differences  among  individual  schemes  (Table  5)  can  also  be  attributed  to  the 
initiatives’  purpose  and  the  environment  in  which  they  were  designed.  While  process  measures 
account  for  the  majority  of  indicators  across  all  initiatives,  the  ratio  of  outcome  and  structure 
measurement  criteria  varies  considerably.  MHS  initiatives  rely  more  heavily  on  compliance-type 
criteria  indicative  of  structure  measures.  These  measures  are  reflective  of  a  management 
environment  governed  primarily  by  mission  completion  versus  financial  incentive. 

Conversely,  outcomes-related  measures  (mostly  in  the  form  of  frequencies  for  select 
diagnoses  and  common  hospital-based  measures  for  length  of  stay  and  inpatient  admissions)  are 
more  common  in  the  civilian  initiatives.  These  measures  reflect  civilian  healthcare 
organizations’  history  of  managing  operations  and  costs  largely  through  utilization  control. 

While  all  six  initiatives  emphasize  measurement  in  the  quality  domain,  significant 
differences  in  the  categories  of  financial  performance  and  customer  satisfaction  were  observed. 

Specifically,  HEDIS  and  the  MHS  PRC  have  few  financial  measures.  HEDIS  is  intended 
for  external  customer  use  versus  internal  management  use;  therefore,  few  financial  details  are 
tracked  and  reported.  Similarly,  the  MHS  PRC  takes  a  DoD  system-wide  view  and  measures  are 
designed  to  communicate  macro-level  performance.  On  the  other  hand,  both  CHAMP  and 
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HCAB  place  significance  emphasis  on  financial  measures.  Since  these  initiatives  were 
specifically  designed  to  identify  and  report  high-costs  procedures,  diagnoses,  and  providers, 
greater  emphasis  on  financial  measurement  is  expected. 

Variance  in  the  area  of  customer  satisfaction  also  appears  significant.  Both  military 
initiatives  contain  from  two  to  five  times  as  many  customer  satisfaction  indicators  as  the  major 
civilian  initiatives;  however,  upon  closer  examination,  specific  indicator  construction  accounts 
for  much  of  the  difference.  The  AFMS  PMT,  for  example,  combines  general  customer 
satisfaction  as  well  as  access  to  care  in  this  category.  The  main  difference  is  how  the  measure  is 
defined:  subjectively  (customer  satisfaction)  or  objectively  (quality — access  to  care). 

A  similar  situation  is  apparent  within  the  preventive  and  utilization  categories.  MHS 
measurement  schemes  tend  to  incorporate  utilization  measures  within  preventive  indicators, 
reflecting  more  proactive  monitoring  of  preventive  quality  measures  and  less  emphasis  on 
reporting  traditional  utilization  data. 

Health  plan  executives  interviewed  subjectively  judged  the  adequacy  of  performance 
indicator  sets  (Appendices  C  through  I)  as  adequate.  Of  the  five  specific  performance 
measurement  methodologies  examined,  two  support  report  card-like  measurement  (HEDIS  and 
MHS  PRC),  two  provide  management-grade  health  plan  information  (CRISP  and  AFMS  PMT), 
and  one  facilitates  detailed  financial  performance  management  (CHAMP). 

Interviewees  identified  the  following  Medicare-specific  indicators  as  areas  not 
adequately  addressed:  (a)  complaint/grievance  trends,  (b)  prescription  drug  profile  management, 
(c)  initial  physical  exam  completion/health  risk  appraisal,  (d)  early  high-risk  identification 
protocols,  (e)  detailed  population  risk  comparisons,  and  (f)  case/disease  management  program 
efficacy. 
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Although  each  methodology  included  in  the  study  demonstrates  areas  of  excellence,  none 
were  found  to  be  one-size-fits-all  measurement  systems,  particularly  in  the  context  of  Medicare 
health  plan  operations.  Each  initiative  was  designed  for  a  specific  environment  and  a  specific 
purpose.  The  AFMS  PMT  is  the  nearest  thing  to  an  existing,  self-contained  MHS  measurement 
tool  capable  of  monitoring  Medicare  subvention  health  plan  performance.  Of  the  civilian 
systems  studied,  the  combination  of  CRISP  and  HEDIS  provide  the  best  opportunity  for  health 
plan  performance  information  gathering.  Comparison  of  the  AFMS  PMT  with  CRISP/HEDIS 
combination  is  presented  in  Table  7. 

Primary  differences  are  observed  in  the  areas  of  customer  satisfaction,  prevention,  and 
utilization;  however,  in  combination  the  three  initiatives  complement  each  other.  Weaknesses  in 
one  scheme  are  generally  offset  by  strengths  in  another. 
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Table  7 

Comparison  of  Health  Plan  Performance  Measurement  Data  Elements  from  AFMS  PMT  and 
CRISP/HEDIS  Measurement  Systems 


INDICATOR 

CHARACTERISTIC 

CRISP/HEDIS 

% 

(n=78) 

AFMS  PMT 
% 

_ (0=27) _ 

A 

% 

INDICATOR  CRITERIA 

Process 

87.2 

85.2 

2.0 

Outcome 

9.0 

0.0 

9.0 

Structure 

3.8 

14.8 

11.0 

INDICATOR  CATEGORY 

Financial  Performance 

7.7 

14.8 

7.1 

Customer  Satisfaction 

3.8 

25.9 

22.1 

Quality 

48.8 

44.4 

4.4 

Quality — General 

9.0 

3.7 

5.3 

Quality — Preventive 

15.4 

33.3 

17.9 

Quality — Outcome 

9.0 

0.0 

9.0 

Quality — Disease  Mgt 

6.4 

7.4 

1.0 

Quality — Access  to  Care 

9.0 

0.0 

9.0 

Community  Service 

1.3 

0.0 

1.3 

Utilization 

25.6 

0.0 

25.6 

Enrollment 

5.1 

7.4 

2.3 

Population 

3.8 

3.7 

0.1 

Other 

3.8 

3.7 

0.1 
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Discussion 

The  three  primary  performance  measurement  methodologies  considered  (AMFS  PMT, 
CRISP,  and  HEDIS)  were  developed  with  different  purposes  and  perspectives.  The  uniqueness 
of  an  MHS  Medicare  Subvention  Project  environment  makes  it  unlikely  that  any  one  of  these 
initiatives  would  be  sufficient  in  subvention  performance  measurement.  However,  the 
combination  of  the  three  models  holds  promise  as  a  model  performance  measurement  system. 

The  AFMS  PMT  examines  27  performance  indicators  in  three  categories:  technical 
outcomes,  customer  satisfaction,  and  financial  performance  (detailed  at  Appendix  C).  The 
AFMS  views  the  PMT  as  “a  critically  important  first  step”  (Foster,  1997,  p.  10).  The  tool 
identifies  standardized  metrics,  specifies  the  methodology,  provides  the  AFMS  with  an 
opportunity  to  learn  the  principles  and  practices  of  integrating  existing  systems  and  processes  to 
improve  the  quality  of  care.  “The  desired  end  state  is  an  accountable  health  plan  with  data-driven 
quality  improvement  ‘built-in,’  not  ‘added  on’”  (AFMS,  1998,  p.  3.). 

CRISP  began  with  91  proposed  performance  indicators.  The  actual  list  of  indicators  used 
in  the  project  was  first  reduced  to  33,  based  on  the  availability  of  individual  data  elements,  then 
to  18,  as  a  result  of  prioritization  by  participating  health  systems  (Nerenz,  Zajac,  and  Rosman, 
1993).  A  description  of  the  CRISP  performance  indicators  is  at  Appendix  E.  CRISP  is  unique  in 
that  it  focuses  on  performance  measures  across  integrated  healthcare  delivery  systems  rather 
than  within  individual  delivery  systems.  It  approaches  this  primarily  by  looking  for  ways  to 
measure  episodes  of  care  that  cross  traditional  organizational  boundaries  and  to  evaluate  the 
effect  (outcomes)  systems  have  on  specific  populations. 

According  to  CRISP,  indicators  that  show  promise  in  quantifying  clinical  outcomes 
include  population  health,  quality  of  care,  episode  characteristics,  episode  prevention,  efficiency, 
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and  satisfaction  (Bergman,  1994).  Care  must  be  taken  that  purported  outcomes  measures  are 
indeed  feasible  as  well  as  meaningful. 

The  newest  version  of  HEDIS,  HEDIS  1999,  includes  a  primary  reporting  set  of  52 
measures  organized  in  eight  general  areas.  Details  for  HEDIS  3.0,  HEDIS  1999,  as  well  as  a 
description  of  the  proposed  changes  between  the  two  are  provided  at  Appendix  G.  HEDIS  is 
sponsored,  supported,  and  maintained  by  the  NCQA — a  not-for-profit  organization  committed  to 
evaluating  and  publicly  reporting  on  the  quality  of  managed  care  plans  (NCQA,  1998).  MCO 
reporting  to  HEDIS  is  voluntary  and  self-reported.  While  MCOs  may  withhold  data  on  specific 
measures  that  they  feel  may  compromise  their  market  competitiveness,  HEDIS’  goal  of  public 
accountability  is  an  important  aspect  of  a  comprehensive  performance  measurement  system 
(Moore,  1997).  HCFA’s  emphasis  on  HEDIS  makes  it  an  indispensable  part  of  a  Medicare 
performance  measurement  model. 

Although  the  comparison  of  the  AFMS  PMT  with  the  CRISP/HEDIS  combination  in 
Table  7  demonstrates  variability  in  both  indicator  criteria  and  category  makeup,  closer 
examination  of  specific  indicator  specification  closes  the  gap  considerably.  For  example,  the 
variance  of  the  two  categories  showing  a  delta  of  over  20  percent  (customer  satisfaction,  and 
utilization)  is  a  result  of  indicator  construction.  The  AFMS  PMT  collects  access  to  care  and 
utilization  data  as  part  of  the  customer  satisfaction  domain.  Conversely,  both  HEDIS  and  CRISP 
tend  to  quantify  utilization  in  terms  of  usage  patterns  and  specific  rates  for  selected  diagnoses. 

A  similar  situation  is  demonstrated  within  the  overall  quality  domain.  On  the  surface  it 
appears  that  the  AFMS  PMT  ignores  outcome  measures.  However,  further  investigation 
illustrates  that  outcomes,  while  not  explicitly  defined,  are  embodied  in  preventive  measures. 
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Emphasis  on  the  process  of  proactive  prevention  versus  retrospective  outcome  measurement  is 
one  of  the  AFMS  PMT’s  strengths  within  the  context  of  Medicare. 

The  construction  of  a  model  Medicare  subvention  performance  measurement  tool  draws 
on  the  strengths  of  all  three  initiatives.  The  AFMS  PMT  provides  the  operational  framework  to 
measure  an  MHS  accountable  health  plan.  CRISP  adds  invaluable  clinical  outcomes 
measurement  to  refine  the  delivery  of  healthcare.  Finally,  HEDIS  provides  a  necessary  measure 
of  public  accountability  by  holding  MHS’s  “feet  to  the  fire.”  While  the  AFMS  PMT  and  CRISP 
provide  the  means  to  monitor  health  plan  performance,  HEDIS  provides  the  means  to  benchmark 
and  report  performance  improvement — both  necessary  ingredients  in  the  scoring  of  the  Medicare 
subvention  demonstration  project.  The  proposed  Medicare  Subvention  Demonstration  Project 
performance  measurement  scheme  with  minimum  performance  indicators  is  provided  (Table  8). 

The  model  proposes  a  structural  framework  for  MHS  Medicare  health  plan  performance 
measurement.  Additional  research  is  required  in  order  to  refine  specific  performance  indicator 
definition  and  threshold  development.  Appropriateness  of  clinical  outcomes  measures  is  best 
determined  by  clinicians.  Additionally,  performance  measurement  systems  must  be  constructed 
as  dynamic,  flexible  “works-in-progress”  capable  of  being  easily  modified  and  adapted  to 
changing  environments. 

Although  elements  from  these  three  initiatives  provide  great  promise  in  MHS  health  plan 
performance  measurement,  several  key  issues  surrounding  the  demonstration  project  remain. 
Benchmarking  has  been  a  popular  “buzz  word”  in  quality  improvement  circles  over  the  last 
decade;  however,  herein  lies  a  potential  trap.  As  pointed  out  by  Mason  and  Dawson  (1998),  it  is 
critical  to  recognize  the  limitations  of  benchmarking.  They  point  out  that  benchmarking  may  be 
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“measuring  the  wrong  way  to  do  the  right  thing.”  All  processes  deserve  a  critical  review  to 
ensure  efficiency  and  effectiveness. 

Two  other  potential  problems  exist  with  benchmarking,  especially  at  the  national  level. 
As  touched  on  earlier  in  the  discussion  of  HEDIS,  many  measurements  are  viewed  as 
proprietary.  Even  those  measures  that  are  in  the  public  domain  may  have  originated  from 
proprietary  systems.  Unless  the  specifications  for  performance  indicators  are  defined, 
benchmarking  looses  its  utility.  Performance  indicators  that  are  useful  as  benchmarks  must  detail 
how  the  measure  is  calculated — where  the  numerator  and  denominator  come  from  and  what 
assumptions  are  built  into  the  formula.  Proprietary  systems  are  not  always  willing  to  divulge  the 
algorithms  used  by  their  programs  (Mason  and  Dawson,  1998). 

Data  standardization  is  another  key  issue.  With  hundreds  of  medical  information  systems 
on  the  market,  terminology  differences  abound.  BJC  Health  System’s  Project  Spectrum,  one  of 
the  most  progressive  medical  information  projects  in  the  country,  demonstrates  the  difficulty  of 
terminology  standardization.  Even  within  a  single  health  system,  the  most  complicated  aspect  of 
engineering  a  common  hospital  information  system  backbone  that  connects  various  hospital, 
diagnostic,  and  provider  information  systems  was  developing  a  data  dictionary  capable  of 
reconciling  the  various  system  terminology  differences  (D.  A.  Streibig,  personal  communication, 
April  16,  1998). 

Fortunately,  MHS  medical  information  system  standardization  has  been  a  priority  since 
the  early  1990s  (Hart  and  Connors,  1996).  Standardization  issues  still  exist  between  the  various 
military  departments;  however,  the  obstacle  within  the  MHS  should  be  more  easily  overcome 
than  within  the  industry  as  a  whole.  Full,  DoD  wide  CEIS  implementation  is  expected  to  be 
complete  by  Fiscal  Year  1999  and  will  solve  most  terminology  issues  (CEIS  (EA),  1997). 
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Table  8 


Proposed  Model  Medicare  Subvention  Health  Plan  Performance  Indicator  Specifications 


EVALUATION  AREA 

POSSIBLE  INDICATOR 

Technical  Outcome  Metrics  1 

Population  Health 

- j - - - j - 

General  health  index 

Disease-specific  outcomes  of  care  (using  SF-36 
and  disease-specific  function  and  symptom  scale) 

Baseline  physical  exams  (chronic 
condition  identification) 4 

%  of  new  Medicare  enrollees  who  receive  initial 
physical  exam/health  risk  screening  within  the 
first  3  months 

Identification  of  Enrolled  Population 

Medicare  DEERS  enrolled 1 

%  of  Medicare-eligible  members  residing  w/in  50 
miles  of  the  base  where  the  MTF  is  physically 
located  who  are  enrolled  in  DEERS 

DEERS  data  quality 1 

%  of  MTF  enrollees  in  TRICARE  Prime  that  are 
identified  in  the  DEERS  database  with  the 
appropriate  code  in  the  provider  code  field 

Epidemiological  Assessment 

Prime  HEAR 1 

%  of  TRICARE  Prime  enrolled  to  the  MTF 
(Medicare-eligible)  with  a  completed  Health 
Enrollment  Assessment  Review  (HEAR)  within  3 
months  of  the  last  day  of  the  month  enrolled 

Managed  Care:  Clinical  Preventive  Svc 

Breast  cancer  screening  ’ 

%  Women  up  to  age  69  with  Mammogram  in  the 
last  2  years 5 

Mammogram  result 1 

%  of  mammograms  at  the  MTF  for  which  the 
patient  was  informed  of  the  results  within  14  days 
of  the  date  the  test  was  performed 

Cervical  screening 2,3 

%  Women  age  up  to  age  64  with  Pap  Smear  in  the 
last  3  years 5 

Pap  smear  result 1 

%  of  all  Pap  smear  tests  reported  to  providers 
during  a  given  month  for  which  the  patient  was 
informed  of  results  within  14  days  of  the  date  of 
the  test 

Prostate  cancer  screening  4 

%  Men  who  have  had  prostate  exam/blood  test  in 
the  last  2  years 5 

Prostate  exam  results  4 

%  of  prostate  exams  at  the  MTF  for  which  the 
patient  was  informed  of  the  results  within  14  days 
of  the  date  the  test  was  performed 

Cholesterol  screening  3 

%  Patients  who  have  had  cholesterol  checked  in 
the  last  2  years 5 

Cholesterol  results 3 

%  of  patients  with  elevated  HDL  levels  who 
received  wellness  counseling  within  14  days  of  the 
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EVALUATION  AREA 


Eye  exams  for  people  with  diabetes 


Immunization  status  1,3 


Episode  Prevention 


Hospital  admissions 3 


Preventable  acute  episode  trend 


Illness-based  medical  care  episodes 


New  breast  cancer  cases 


Episode  Characteristics 


Services  per  episode 2 


Managed  Care:  Case/Disease 
Management 


Case  management  activity 


Diabetic  management  *’ 


Asthma  management  '* 


Cardiovascular  event  management 


Depression  Management 1,3 


Pharmacy  Services  Management 


Prescription  drug  consultation J 


Outpatient  drug  utilization 3 


Quality  of  Care 2 


Customer  Satisfaction  Metrics  1 


Access  *’ _ 

Resources  1 _ 

Waiting  time  at  appointment 1 


Quality  1,2,3 _ _ 

Health  advice — annual  *’ 3 


POSSIBLE  INDICATOR 


date  the  test  was  performed _ 

%  Diabetic  patients  with  eye  exam  within  the  last 
2  years 5 


%  of  Medicare  enrollees  who  are  current  on  their, 
Tetanus  and  Influenza  immunizations 


Hospital  Admissions  per  member  per  year 


Frequency  of  preventable  acute  episodes  within 
chronic  conditions  (emergency  room  visits  for 
patients  with  asthma) 


Number  of  illness-based  medical  care  episodes  per 
member  per  year 


Percentage  of  new  breast  cancer  cases  classified  as 
“advanced” _ 


Hospital  days  per  1,000  members 


Number  of  services  per  episode  (for  selected 
diagnoses 


Number  of  members  in  active  case  management 
by  chronic  condition  (CHF,  diabetes,  asthma,  HIV, 
oncology 


%  of  diabetics  tested  for  glycohemoglobin/ 
hemoglobin  A1C  as  ordered  by  an  MTF  provider 
during  the  past  12  months  5  _ 


Ratio  of  asthmatic  patients  who  are  on  preventive 
(or  controller)  asthma  medications  _ 


Beta  blocker  treatment  after  a  heart  attack 


Prescription  compliance  ratio  (number  of 
consecutive  monthly  prescriptions  filled  divided 
bv  the  number  of  prescriptions  written) 


%  New  members  who  received  prescription  drug 

consultation  within  3  months  of  enrollment _ 

Trends  in  outpatient  drug  utilization 


Compliance  with  standard  care  patterns  (for 
selected  diagnoses) 5 


Satisfaction  with  access  to  appointments _ 

Satisfaction  with  access  to  system  resources _ 

TRICARE  encounter  waiting  time  standards 


Satisfaction  with  technical  quality _ 

Have  healthcare  providers  given  advice  on  ways  to 
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EVALUATION  AREA 

POSSIBLE  INDICATOR 

stay  healthy 

Waiting  time  for  appointment 1 

Number  of  days  between  day  appointment  was 
made  and  the  day  of  the  appointment 

Waiting  time  expectation  1 

Customer  expectation  of  number  of  days  to  wait 
for  an  appointment  to  occur 

Ease  in  making  an  appointment 1 

Ease  of  making  an  appointment  by  phone 

Enrollment 1,3 

Enrollment  in  TRICARE  Prime 

Vote  with  their  feet 1 

Health  Advice — Outpatient 1 

Satisfaction  with  advice  received  about  ways  to 
avoid  illness  and  stay  healthy 

Financial  Metrics  1 

Spendline  1 

Spendline  (budget  targets  versus  budget 
obligations) 

Economic  value  (revenue) 1 

Economic  value  (market  value  that  approximates 
opportunity  revenue) 

Operating  expense  1 

Operating  expense 

Throughput 1 

Throughput  (Economic  value  -  Operating 
expense) 

High-Occurrence/High-Cost  DRGs  *’ 2 

High-Occurrence/High-Cost  DRGs 

Note.  From  a  combination  of: 

1 FY  98  Air  Force  Medical  Service  Performance  Measurement  Tool  (Version  2.0),  by  the  Office 
for  Prevention  and  Health  Services  Assessment,  Air  Force  Medical  Service,  1998,  Brooks  Air 
Force  Base,  TX:  Author; 

2  “Consortium  Research  on  Indicators  of  System  Performance  (CRISP),  by  D.  R.  Nerenz,  B.  M. 
Zajac,  and  H.  S.  Rosman,  1993,  The  Joint  Commission  Journal  on  Quality  Improvement,  19, 

582; 

3NCQA  HEDIS  1999  Draft  Document,  by  the  National  Committee  on  Quality  Assurance,  1998, 
[Text  file].  URL  ftp://info.ncqa.org/ftp/heddraft.doc;  and 
4  Various  interview  sources. 

Specific  performance  indicator  specifications  (i.e.  %  of  women  with  mammograms  within  last  2 
years)  were  extracted  from  one  of  the  sources  cited  and  are  included  at  this  point  for 
demonstration  purposes.  Additional  refinement  of  measurement  levels  must  be  accomplished  in 
conjunction  with  clinical  personnel. 


The  final  issue,  while  not  entirely  within  the  scope  of  performance  measurement, 
significantly  impacts  the  design  and  operation  of  measurement  systems.  All  healthcare 
executives  interviewed  as  part  of  this  project  stressed  the  importance  of  long-term  versus  short¬ 
term  measurement,  wellness  and  prevention  versus  treatment  and  reaction,  and  primary  care 
investment  versus  specialty  care  expenditure.  The  problem  lies  in  the  fact  that  the  Medicare 
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Subvention  Demonstration  Project  will  be  evaluated  over  a  three-year  term.  The  first  year 
consists  of  health  plan  definition  and  HCFA  approval,  leaving  only  two  years  to  actively  enroll 
and  managed  a  Medicare  population.  Do  project  managers  operationalize  subvention  with  long¬ 
term,  preventive,  primary  care  processes  in  an  attempt  to  raise  the  population’s  health  status  and 
impact  hospital  expenses  in  the  out  years?  Or  do  they  program  short-term  expenses  in 
anticipation  of  being  able  to  demonstrate  to  HCFA  that  they  can  provide  “cheaper”  care  at  the 
end  of  the  demonstration  period?  The  performance  measurement  system  that  supports  the 
preferred  course  of  action  must  define  indicators  based  on  that  course  of  action — courses  of 
action  that  are  seemingly  diametrically  opposed. 

Ultimate  success  in  Medicare  Subvention  will  be  found  in  the  successful  execution  of  the 
Medicare  health  plan  in  two  key  areas.  First  of  all,  HCFA  provides  an  open-book  test  for 
Medicare  HMO  operations — structure  and  process  aspects  of  health  plan  performance 
measurement.  The  MHS  is  comfortable  and  well  practiced  in  the  area  of  compliance  inspection 
since  MTFs  participate  in  the  same  accreditation  programs  as  civilian  institutions  in  addition  to 
running  their  own  version  of  inspection  through  the  Inspector  General  function. 

The  second  key  area,  managing  population  health  at  the  right  cost,  is  the  area  likely  to 
“make  or  break”  MHS  operations.  Early  identification  of  chronic  members,  as  well  as  potentially 
chronic  members,  followed  by  proactive  prevention  versus  reactive  intervention  is  the  shortest 
path  to  providing  appropriate  health  care  at  acceptable  costs.  The  likelihood  of  success  without 
effective  performance  measurement  systems  is  low. 

Either  military  or  civilian  health  professionals  can  use  results  of  the  study  to  define  or 
refine  health  plan  performance  measurement  systems.  Specifically,  this  study  can  aid  in  the 
design  and  evaluation  of  Medicare  managed  care  service  line  products.  Medicare  Subvention 
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Project  Officers  and  MTF  Commanders  will  be  able  to  track  internal  plan  performance  as  the 
demonstration  project  progresses.  Ultimate  beneficiaries  of  an  efficient,  cost-effective,  high 
quality  Medicare  health  plan  include  HCFA,  DoD,  and  countless  Medicare-eligible  retirees. 

Sound  performance  measurement  techniques  can  also  be  used  to  support  business  case 
analyses  as  well  as  make-versus-buy  decisions.  As  the  healthcare  industry  further  evolves,  the 
model  may  be  used  to  evaluate  existing  and  proposed  programs.  Additionally,  resulting 
performance  measurement  schemes  can  be  extrapolated  for  use  with  any  healthcare  insurance  or 
healthcare  delivery  product. 


Conclusions  and  Recommendations 

With  some  modification  MHS  performance  measurement  systems  appear  to  be  adequate 
to  evaluate  Medicare  Subvention  Demonstration  Project  success. 

This  study  proposes  a  best  case  model  of  health  plan  performance  indicators.  A  question 
of  significance,  but  beyond  the  scope  of  this  study  and  perhaps  an  area  of  subsequent 
investigation,  is  the  accuracy  of  data  used  in  the  proposed  model. 

Clearly,  more  similarities  than  differences  exist  between  civilian  and  military  health 
systems.  Many  of  the  data  elements  captured  in  each  system  are  the  same;  however,  the  degree 
which  these  elements  are  used  and  how  they  are  used  in  performance  measurement  differs 
significantly.  MHS’s  relatively  recent  interest  in  accurately  pricing  the  provision  of  healthcare 
services  (TMA,  1998),  results  in  a  steep  learning  curve.  Direct  comparison  of  MHS  and  civilian 
Medicare  health  plan  performance,  particularly  in  the  area  of  cost,  may  not  be  practical  at  this 
time. 

The  construction  of  an  ideal  Medicare  subvention  performance  measurement  tool  draws 
on  the  strengths  of  three  initiatives:  AFMS  PMT,  CRISP,  and  HEDIS.  AFMS  PMT  and  CRISP 
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provide  the  means  to  monitor  health  plan  performance  and  HEDIS  provides  the  means  to 
benchmark  and  report  health  plan  performance  improvement. 

Specific  recommendations  include: 

(a)  MHS  embrace  the  AFMS  PMT  as  the  MHS  standard 

(b)  Enhance  the  AFMS  PMT  with  clinical  outcomes  measures  from  CRISP  and  tailor 

measures  for  a  Medicare  population 

(c)  Incorporate  HEDIS  in  routine  MHS  health  plan  performance  measurement 

(d)  MHS  consider  participation  in  the  Consortium  Research  on  Indicators  of  System 

Performance  outcomes  measurement  research 

(e)  Challenge  the  MHS  to  take  a  long-term,  patient-centered  approach  to  Medicare 

subvention  and  for  the  GAO  to  score  the  demonstration  project  as  such 

At  a  minimum,  the  proposed  performance  measurement  model  at  Table  8  provides  a 
conceptual  framework  for  the  development  of  organization-specific  measurement  tools.  Final 
definition  of  individual  performance  indicators  is  a  matter  of  coordination  among  the 
organization’s  management,  clinical,  and  information  system  elements.  Management’s  function 
is  to  ensure  resulting  performance  indicators  are  linked  to  the  organization’s  mission.  Clinical 
expertise  is  paramount  to  linking  health  plan  performance  measurement  to  meaningful  outcomes 
measurement.  Finally,  information  system  personnel  are  vital  in  making  the  process 
operationally  possible.  A  well  conceived,  thoughtfully  designed,  technically  sound  performance 
measurement  system  arms  health  plan  managers  with  the  tools  needed  to  serve  both  the 
organization  and  the  customer. 

Medicare  managed  care  success  is  dependent  on  two  primary  goals:  reduction  of 
inpatient  utilization  and  managing  care  along  the  continuum  of  services  (beginning  with  early 
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detection  and  prevention  of  chronic  conditions).  The  MHS  enjoys  two  distinct  advantages  that 
serve  as  a  bridge  to  this  success.  Health  and  demographic  information  is  available  for  all 
potential  members  and  the  MHS  possesses  control  over  the  majority  of  the  care  continuum. 
Timely,  effective  performance  measurement  is  critical  in  attaining  Medicare  risk  plan  success. 
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Appendix  A 


Acronym  List 

AFMS 

Air  Force  Medical  Service 

CAHPS 

Consumer  Assessments  of  Health  Plans  Study 

CEIS 

Corporate  Executive  Information  System 

CEIS  (EA) 

Corporate  Executive  Information  System  Executive  Agent 

CHP 

Comprehensive  Health  Planning 

CHAMP 

Comprehensive  Healthcare  Analysis  and  Management  Program 

CRISP 

Consortium  Research  on  Indicators  of  System  Performance 

DHP 

Defense  Health  Plan 

DoD 

Department  of  Defense 

DRG 

Diagnostic  Related  Group 

EBC 

Enrolment-Based  Capitation 

FAcct 

Foundation  for  Accountability 

GAO 

General  Accounting  Office 

HCAB 

Health  Care  Advisory  Board 

HCFA 

Health  Care  Financing  Administration 

HEDIS 

Health  Plan  Employer  Data  and  Information  Set 

HFHS 

Henry  Ford  Health  System 

HMO 

Health  Maintenance  Organization 

JCAHO 

Joint  Commission  on  Accreditation  of  Healthcare  Organizations 

LOE 

Level  of  Effort 

MCO 

Managed  Care  Organization 
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MEPRS 

Medical  Expense  and  Performance  Reporting  System 

MRSP 

Medical  Readiness  Strategic  Plan 

MHS 

Military  Health  Service 

MHSS 

Military  Health  Service  System 

MSO 

Medical  Service  Organization 

MTF 

Military  Treatment  Facility 

NCQA 

National  Committee  for  Quality  Assurance 

OSD  (HA) 

Office  of  the  Assistant  Secretary  of  Defense  for  Health  Affairs 

PCM 

Primary  Care  Manager 

PMT 

Performance  Measurement  Tool  (AFMS  PMT) 

PRC 

Performance  Report  Card  (MHS  PRC) 

RMP 

Regional  Medical  Programs 

TMA 

TRICARE  Management  Activity 

WWW 

World  Wide  Web 
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Appendix  B 


MHS  Medical  Information  Systems 


ACRONYM 

TITLE 

DESCRIPTION 

ADS-OMR 

Ambulatory  Data  System 
Optical  Mark-Sense 

Reader 

ADS  is  the  interim  solution  to  outpatient 
encounter  information;  ADS  will  replaced  by  the 
capabilities  of  CIW  when  it  comes  online 

CDIS 

CHAMPUS  Detail 
Information  System 

Supports  online,  near  real-  time  accessing  and 
retrieval  of  individual  detailed  CHAMPUS 
information 

CEIS 

Corporate  Executive 
Information  System 

A  decision  support  system  focused  on  the 
collection,  integration,  Information  organization, 
and  display  of  the  information,  primarily  drawn 
from  other  needed  by  MTF  commanders  and 
other  managers,  to  make  more  effective 
management  decisions 

CHCS 

Composite  Health  Care 
System 

CHCS  is  a  primary  source  data  collection  system 
in  the  MHS,  providing  data  for  the  PAS,  PAD, 
LAB,  RAD,  Clinical  PHARM,  and  Nursing; 
CHCS  creates  a  portion  of  the  electronic  record 
data  that  exists  today 

CIS 

Clinical  Information 

System 

The  patient-  focused  clinical  arm  of  the  managed 
care  initiative  combining  the  efforts  of  the 
patient-focused  (Nursing)  and  special  care  units 
through  an  interdisciplinary  covering  critical 
paths,  and  clinical  management 

CIW 

Clinical  Workstation 

Provides  a  seamless  integrated  clinical  within  a 
subset  of  the  functionality  encompassed  by  the 
Provider  Workstation  (PWS),  Ambulatory  Data 
Collection  system  (ADCS),  Inpatient  Order 

Entry  (IPOE),  Outpatient  Order  Entry  (OPOE), 
CIS,  and  MDIS 

DMHRS 

Defense  Medical  Resource 
System 

Provides  automated  system  support  calculating 
military  and  civilian  labor  time  and  cost  for  DoD 
health  care  activities 

DMLSS 

Defense  Medical  Standard 
Support 

Provides  automated  support  for  the  of  facilities, 
technology,  equipment,  supplies,  and  services 
within  MHSS  in  peacetime  and  wartime 

MEPRS/ 

EAS  IV 

Medical  Expense  and 
Performance  Reporting 
System/Expense 

Assignment  System  (v.  IV) 

A  proposed  system  to  provide  support  for 
standardized  reporting  of  expenses,  manpower, 
and  workload  data  at  the  work  center  level 
within  DoD,  facilities 

PWS 

Provider  Workstation 

The  provider  interface  to  CHCS  for  placing 
orders;  PWS  is  the  core  capability  for  CIW  and 
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ACRONYM 

TITLE 

DESCRIPTION 

has  been  absorbed  into  the  CIW  program 

SPS/EPS 

Space  Planning 
System/Equipment 

Planning  System 

Provides  automated  estimate  of  the  size  and 
functional  requirement  for  health  care  and 
medical  research  facilities,  using  DoD 
established  medical  space  planning  criteria 

TPOCS 

Third  Party  Outpatient 
Collection  System 

Is  a  DoD  standard  system  that  assists  MTFs  in 
the  collection,  tracking,  and  reporting  of  data 
required  in  the  outpatient  insurance  billing 
process 

Note.  Data  extracted  from  “A  Resource  Decision  Model  for  Military  Hospitals,”  by  S.  E.  Hart 
and  R.  E.  Connors,  1996,  Military  Medicine,  161  [WWW  document].  URL  http://hawww.ha.osd. 
mil/main/hartdoc.html  and  “P3  System  Descriptions,”  by  the  Office  of  the  Assistant  Secretary  of 
Defense,  Health  Affairs,  1998,  IPPSRS  P3  System  Descriptions  [WWW  document].  URL 
http://www.ha.osd.mil/dmim/imit/p3sysdes.html. 
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Appendix  C 


Air  Force  Medical  Service  Performance  Measurement  Tool  (AFMS  PMT)  Indicator 
Specifications  (1997) 


EVALUATION  AREA 


Technical  Outcome  Metrics _ 

Identification  of  Enrolled  Population 


Active  duty  DEERS  enrolled 


DEERS  data  quality 


Epidemiological  Assessment 


Prime  HEAR 


Managed  Care:  Clinical  Preventive  Svc 


Mammogram  result 


Pap  smear  result 


Managed  Care:  Disease  Management 


Diabetic  management 


Asthma  management 


Operational  Metrics 
Immunization  status 


Dental  status 


INDICATOR 


%  of  ADAF  members  assigned  to  the  base  where 
the  MTF  is  physically  located  who  are  enrolled  in 
DEERS  _ 


%  of  MTF  enrollees  in  TRICARE  Prime  that  are 
identified  in  the  DEERS  database  with  the 
appropriate  code  in  the  provider  code  field 


%  of  TRICARE  Prime  enrolled  to  the  MTF  (age 
17  and  older)  with  a  completed  Health  Enrollment 
Assessment  Review  (HEAR)  within  3  months  of 
the  last  day  of  the  month  enrolled  _ 


%  of  mammograms  at  the  MTF  for  which  the 
patient  was  informed  of  the  results  within  14  days 

of  the  date  the  test  was  performed _ 

%  of  all  Pap  smear  tests  reported  to  providers 
during  a  given  month  for  which  the  patient  was 
informed  of  results  within  14  days  of  the  date  of 
the  test  _ 


%  of  diabetics  >18  years  old  tested  for 
glycohemoglobin/hemoglobin  A1C  as  ordered  by 

an  MTF  provider  during  the  past  12  months _ 

Ratio  of  asthmatic  patients  age  39  or  less  who  are 
on  preventive  (or  controller)  asthma  medications 


%  of  active  duty  members  assigned  to  the  base 
where  the  MTF  is  physically  located  who  are 
current  on  their  Hepatitis  A,  Tetanus,  and 
Influenza  immunizations 


%  of  active  duty  Air  Force  members  on  the  base 
where  the  MTF  is  physically  located  who  are  in 
Dental  Class  1  or  2  _ __ 


%  of  active  duty  Air  Force  members  on  the 
adjusted  base  alpha  roster  who  tested  and  met 
fitness  standards  according  to  the  FIT 
management  database 
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EVALUATION  AREA 

INDICATOR 

PHA  completion 

Health  assessment  of  the  deployable  force 

Mission  Availability 

Lost  time  due  to  medical  events 

Mission  availability:  Medically  related  lost  duty 
days 

Customer  Satisfaction  Metrics 

Access 

Satisfaction  with  access  to  appointments 

Resources 

Satisfaction  with  access  to  system  resources 

Waiting  time  at  appointment 

TRICARE  encounter  waiting  time  standards 

Quality 

Satisfaction  with  technical  quality 

Health  advice  —  annual 

Have  healthcare  providers  given  advice  on  ways  to 
stay  healthy 

Waiting  time  for  appointment 

Number  of  days  between  day  appointment  was 
made  and  the  day  of  the  appointment 

Waiting  time  expectation 

Customer  expectation  of  number  of  days  to  wait 
for  an  appointment  to  occur 

Ease  in  making  an  appointment 

Ease  of  making  an  appointment  by  phone 

Enrollment 

Enrollment  in  TRICARE  Prime 

Vote  with  their  feet 

(Member  turnover) 

Health  Advice— Outpatient 

Satisfaction  with  advice  received  about  ways  to 
avoid  illness  and  stay  healthy 

Financial  Metrics 

Spendline 

Spendline 

Economic  value  (revenue) 

Economic  value  (revenue) 

Operating  expense 

Operating  expense 

Throughput 

Throughput 

Note.  From  FY  98  Air  Force  Medical  Service  Performance  Measurement  Tool  (Version  2.0),  by 
the  Office  for  Prevention  and  Health  Services  Assessment,  Air  Force  Medical  Service,  1998, 
Brooks  Air  Force  Base,  TX:  Author. 
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Appendix  D 

MHS  Performance  Report  Card  Indicator  Specifications  (1997) 


PERFORMANCE  MEASURE 

DESCRIPTION 

Access 

Satisfaction  with  Access  to 
Appointments 

%  Military  MTF  users  rating  4  access  areas  including 
appointments  and  office  waiting,  rating  between  good  and 
excellent 

Satisfaction  with  Access  to 

System  Resources 

%  Military  MTF  users  rating  2  access  areas,  convenience 
and  access  to  a  specialist,  rating  between  good  and  excellent 

%  Meeting  Appointment  Waiting 
Standards 

%  Military  MTF  users  waiting  no  more  than  one  week  from 
scheduling  appointment  to  seeing  provider  for  an  acute 
illness  or  injury 

Prime  AD  Enrollment  Rates 

%  AD  enrolled  and  assigned  a  primary  care  manager 

Quality 

Medical  Readiness  Trained  & 
Certified 

%  AD  privileged  providers  assigned  to  a  mobility  platform 
with  sustained  medical  readiness  training  verified  by  the 
commander 

Dental  Readiness 

%  AD  in  Dental  Class  1&  2 

Satisfaction  with  Quality 

%  Military  MTF  users  rating  5  quality  areas  including, 
thoroughness  of  treatment,  final  results,  and  overall  quality 
and  service,  rating  between  good  and  excellent 

%  Women  Age  21-64  w/Pap 

Smear  (Last  3  Yrs) 

%  Women  age  21  -  64  with  Pap  Smear  in  the  last  3  years 

%  Women  Age  52-69 
w/Mammogram  (Last  2  Yrs) 

%  Women  age  52  -  69  with  Mammogram  in  the  last  2  years 

%  Pop  w/Cholesterol  Screen  (Last 

5  Yrs) 

%  Patients  who  have  had  cholesterol  checked  in  the  last  5 
years 

Childhood  Immunization  Rates 

%  Children  at  age  2  that  have  completed  basic 
immunization  series 

JCAHO  Grid  Scores 

Latest  JCAHO  Grid  Score 

JCAHO  Accreditation 

Latest  accreditation  status 

Utilization 

AD  Preventable  Admission  Rates 
Chronic  Obstructive 

Pulmonary  Disease 

Bacterial  Pneumonia 

Asthma 

Congestive  Heart  Failure 

Angina 

Cellulitis 

Diabetes 

Gastroenteritis 

Kidney/Urinary  Infections 

Measures  AD  admissions  per  1000  for  diagnoses  where 
timely  and  effective  outpatient  care  reduces  risk  of 
hospitalization 
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PERFORMANCE  MEASURE 

DESCRIPTION 

ADFM  Preventable  Admission 
Rates 

Chronic  Obstructive 

Pulmonary  Disease 

Bacterial  Pneumonia 

Asthma 

Congestive  Heart  Failure 

Angina 

Cellulitis 

Diabetes 

Gastroenteritis 

Kidney/Urinary  Infections 

Measures  AD  family  member  (age  18+)  admissions  per 

1000  for  diagnoses  where  timely  and  effective  outpatient 
care  reduces  risk  of  hospitalization 

AD  Bed  Day  Rates 

AD  bed  day  rates  per  1000.  Excludes  major  diagnostic 
categories  relating  to  obstetrics  (MDCs  14  &  15)  and 
mental  health/substance  abuse  (MDCs  19  &  20) 

ADFM  Bed  Day  Rates 

ADFM  bed  day  rates  per  1000.  Excludes  major  diagnostic 
categories  relating  to  obstetrics  (MDCs  14  &  15)  and 
mental  health/substance  abuse  (MDCs  19  &  20) 

Health  Behaviors 

%  AD  Smoked  Last  30  Days 

%  AD  reporting  smoking  any  cigarettes  in  the  past  30  days 

%  AD  Problem  Drinkers 

%  AD  reporting  serious  consequences  associated  w/  alcohol 
in  past  year 

%  AD  Dependent  on  Alcohol 

%  AD  reporting  signs  and  symptoms  of  alcohol  dependence 
(e.g.,  sickness  from  drinking,  becoming  intoxicated, 
shaking  hands,  etc.)  for  more  than  48  days  in  the  last  12 
months 

Health  Status 

Perceived  Physical  Health  -  AD 

Self-reported  health  status  scores  measures  individual 
perception  of  their  physical  health  using  the  Rand  SF-12 

Perceived  Physical  Health  - 
ADFM 

Self-reported  health  status  scores  measures  individual 
perception  of  their  physical  health  using  the  Rand  SF-12 

Perceived  Mental  Health  -  AD 

Self-reported  health  status  scores  measures  individual 
perception  of  their  mental  health  using  the  Rand  SF-12 

Perceived  Mental  Health  -  ADFM 

Self-reported  health  status  scores  measures  individual 
perception  of  their  mental  health  using  the  Rand  SF-12 

Note.  From  “MHS  Performance  Report  Card  Measurement  Definitions,”  by  Health  Services 
Analysis  and  Measurement,  1997,  [WWW  document].  URL  http://www.ha.osd.mil/milonly/ 
reptcard/defsept.htm 
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Appendix  E 


Consortium  Research  on  Indicators  of  System  Performance  (CRISP)  Indicator  Specifications 
(1993)  “  ‘ 


EVALUATION  AREA 

INDICATOR 

TIER  I  Indicators 

Population  Health 

General  health  index  (SF-36) 

Prevention  index  [health  risk  appraisall 

Community  Benefit 

Proportion  of  system  expenses  devoted  to  charity  care 

Quality  of  Care 

Hospital  readmission  rate 

Episode  Prevention 

Hospital  Admissions  per  member  per  year 

Low  birthweight  incidences 

Episode  Characteristics 

Number  of  services  per  episode  (for  selected  diagnoses) 

Hospital  days  per  1,000  members 

Satisfaction 

GHAA  member  satisfaction  survey 

Efficiency 

Debt-service  coverage  ratios 

TIER  II  Indicators 

Population  Health 

Disease-specific  outcomes  of  care  (using  SF-36  and  disease- 
specific  function  and  symptom  scale 

Quality  of  Care 

Compliance  with  standard  care  patterns  (for  selected 
diagnoses) 

Episode  Characteristics 

Redundancy  of  services  provided  within  episodes 

Episode  Prevention 

Frequency  of  preventable  acute  episodes  within  chronic 
conditions  (emergency  room  visits  for  patients  with  asthma) 

Illness-based  medical  care  episodes  per  member  pre  year 

Percentage  of  new  breast  cancer  cases  classified  as  “advanced” 

Note.  From  “Consortium  Research  on  Indicators  of  System  Performance  (CRISP),  by  D.  R. 
Nerenz,  B.  M.  Zajac,  and  H.  S.  Rosman,  1993,  The  Joint  Commission  Journal  on  Quality 
Improvement,  19,  582. 
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Appendix  F 

Comprehensive  Healthcare  Analysis  and  Management  Program  Indicator  Specifications  (1997) 

Expense  and  Utilization 
Claims  payment  summary 
Statistical  trends 

Year-over-year  trends,  year-to-date  analysis 

Plan  Performance  Evaluation 

Changes  in  number  of  enrollees 

Changes  in  number  and  expense  of  high  cost  claimants 

Shift  in  age/sex  composition  of  population 

Clinical  Profiles 

Most  prevalent  and  costly  diagnoses  and  procedures 
Intervention  program  identification 
Utilization  review  effectiveness 

Provider  Analysis 

Comparison  of  top  providers  on  billed,  covered,  and  paid  expenses 
High  expense,  high  utilization,  and  high  price  providers 
Cesarean  section  rates 
Surgery  rates 
Referral  patterns 

Benefit  Modeling 

Distribution  of  expenses  and  services 

Number  of  individuals  incurring  more  than  $30,000  in  the  past  12  months 
Maximum  payment  schedule/utilization  maximum  level  development 

Financial  Management 

Comparative  financial  summary 
Rate  setting 
Reserve  analysis 
Cost  allocation 

Large  Case  Management 
Identification  of  high  expense  claimants 
Target  preventive  intervention  programs 

Enrollment  Analysis 

Enrollment  summaries  by  relation,  tier,  age,  sex,  and  zip  code 

Note.  From  Comprehensive  Healthcare  Analysis  and  Management  Program,  by  W.  M.  Mercer, 
Inc.,  1997,  [Brochure],  New  York:  Author. 
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Appendix  G 


HEDIS  3.0  and  HEDIS  1999  Performance  Indicator  Specifications 


HEDIS  3.0/1998 
Specifications 

HEDIS  1999  Specifications 

HEDIS  1999  Specifications 
(changes  from  HEDIS  3.0) 

EFFECTIVENESS  OF 

CARE 

EFFECTIVENESS  OF 

CARE 

Childhood  Immunization 

Status 

Childhood  Immunization  Status 

Modifications  to  antigen- 
specific  and  combination  rates 
and  CPT  codes. 

Adolescent  Immunization 

Status 

Adolescent  Immunization 

Status 

Modifications  to  CPT  codes 
and  documentation 
requirements. 

Advising  Smokers  to  Quit 

Advising  Smokers  to  Quit 

Flu  Shots  for  Older  Adults 

Flu  Shots  for  Older  Adults 

Breast  Cancer  Screening 

Breast  Cancer  Screening 

Cervical  Cancer  Screening 

Cervical  Cancer  Screening 

Prenatal  Care  in  the  First 
Trimester 

Prenatal  Care  in  the  First 
Trimester 

Low  Birth-Weight  Babies 

Low  Birth-Weight  Babies 

Moratorium  continued. 

Check-Ups  After  Delivery 

Check-Ups  After  Delivery 

Beta  Blocker  Treatment  After 
a  Heart  Attack 

Beta  Blocker  Treatment  After  a 
Heart  Attack 

Clarification  on  prescriptions 
rendered  30  days  prior  to 
admission. 

Eye  Exams  for  People  with 
Diabetes 

Eye  Exams  for  People  with 
Diabetes 

Cholesterol  Management  After 
Acute  Cardiovascular  Events  5 

New  measure. 

The  Health  of  Seniors 

The  Health  of  Seniors 

Follow-Up  After 
Hospitalization  for  Mental 
Illness 

Follow-Up  After 

Hospitalization  for  Mental 
Illness 

Seven-day  ambulatory  or 
day/night  follow-up  added. 

Antidepressant  Management 

New  measure. 

Comprehensive  Diabetes  Care 

Possible  new  measure. 
Ongoing  work  is  underway  to 
develop  the  technical 
specifications. 

ACCESS/AVAILABILITY 
OF  CARE 

ACCESS/AVAILABILITY 

OF  CARE 

Adults’  Access  to 

Preventive/ Ambulatory  Health 
Services 

Adults’  Access  to 

Preventive/ Ambulatory  Health 
Services 

Children’s  Access  to  Primary 

Children’s  Access  to  Primary 

Modifications  to  CPT  codes. 
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HEDIS  3.0/1998  HEDIS  1999  Specifications 

_ Specifications _ 

Care  Providers _ Care  Providers _ 

Availability  of  Primary  Care 

Providers _ _ 

Availability  of  Behavioral 

Health  Care  Providers _ _ 

Availability  of  Obstetrical  and 

Prenatal  Care  Providers _ 

Initiation  of  Prenatal  Care  Initiation  of  Prenatal  Care 

Low  Birth-Weight  Deliveries  Low  Birth-Weight  Deliveries  at 

at  Facilities  for  High-Risk  Facilities  for  High-Risk 

Deliveries  &  Neonates _ Deliveries  &  Neonates _ 

ACCESS/AVAILABILITY  ACCESS/AVAILABILITY 

OF  CARE  (continued) _ OF  CARE  (continued) _ 

Annual  Dental  Visit _ Annual  Dental  Visit _ 

Availability  of  Dentists _ _ 

Availability  of  Language  Availability  of  Language 

Interpretation  Services _ Interpretation  Services _ 

SATISFACTION  WITH  SATISFACTION  WITH 

THE  EXPERIENCE  OF  THE  EXPERIENCE  OF 

CARE _ CARE _ 

HEDIS  Consumer  Survey  HEDIS  Consumer  Survey 

(commercial,  Medicaid) _ (commercial,  Medicaid) _ 

Experiences  with  Children’s  Experiences  with  Children’s 

Care  Survey  (commercial,  Care  Survey  (commercial, 

Medicaid) _ Medicaid) _ 

HEALTH  PLAN  HEALTH  PLAN 

STABILITY _ STABILITY _ 

Disenrollment _ Disenrollment _ 

Provider  Turnover _ Provider  Turnover _ 

Years  in  Business/Total  Years  in  Business/Total 

Membership _ Membership _ 

Indicators  of  Financial  Indicators  of  Financial  Stability 

Stability _ 

USE  OF  SERVICES  _ USE  OF  SERVICES _ 


Frequency  of  Ongoing  Prenatal  Frequency  of  Ongoing  Prenatal 
Care  Care 


HEDIS  1999  Specifications 
(changes  from  HEDIS  3.0) 

Measure  retired. 

Measure  retired. 

Measure  retired. 

Language  clarified. 


New  survey  instrument. 
New  survey  instrument. 


Well-Child  Visits  in  the  First 


Well-Child  Visits  in  the  First 


Medical  record  documentation 
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HEDIS  3.0/1998  HEDIS  1999  Specifications 

_ Specifications _ 

15  Months  of  Life _ 1 5  Months  of  Life _ 

Well-Child  Visits  in  the  Third,  Well-Child  Visits  in  the  Third, 
Fourth,  Fifth  and  Sixth  Year  of  Fourth,  Fifth  and  Sixth  Year  of 

Life _ Life _ 

Adolescent  Well-Care  Visit  Adolescent  Well-Care  Visit 

Frequency  of  Selected  Frequency  of  Selected 

Procedures _ Procedures _ 

Inpatient  Utilization— General  Inpatient  Utilization— General 

Hospital/ Acute  Care _ Hospital/Acute  Care _ 

Ambulatory  Care  Ambulatory  Care 

Inpatient  Utilization-Non-  Inpatient  Utilization-Non- 

Acute  Care _ Acute  Care _ 

Discharge  and  Average  Length  Discharge  and  Average  Length 

of  Stay— Maternity  Care _ of  Stay— Maternity  Care _ 

Cesarean  Section  and  Vaginal  Cesarean  Section  and  Vaginal 
Birth  After  Cesarean  Rate  Birth  After  Cesarean  Rate 

(VBAC-Rate) _ (VBAC-Rate) _ 

Births  and  Average  Length  of  Births  and  Average  Length  of 

Stay,  Newborns _ Stay,  Newborns _ 

Mental  Health  Utilization—  Mental  Health  Utilization- 
Inpatient  Discharges  and  Inpatient  Discharges  and 

Average  Length  of  Stay _ Average  Length  of  Stay _ 

Mental  Health  Utilization—  Mental  Health  Utilization- 
Percentage  of  Members  Percentage  of  Members 

Receiving  Inpatient,  Day/Night  Receiving  Inpatient,  Day/Night 

and  Ambulatory  Services  and  Ambulatory  Services _ 

Readmission  For  Specified 

Mental  Health  Disorders _ 

Chemical  Dependency  Chemical  Dependency 

Utilization-Inpatient  Utilization-Inpatient 

Discharges  and  Average  Discharges  and  Average  Length 

Length  of  Stay _ of  Stay _ 

Chemical  Dependency  Chemical  Dependency 

Utilization— Percentage  of  Utilization— Percentage  of 
Members  Receiving  Inpatient,  Members  Receiving  Inpatient, 
Day/Night  Care  and  Day/Night  Care  and 

Ambulatory  Services _ Ambulatory  Services _ 

Readmission  for  Chemical 

Dependency _ 

Outpatient  Drug  Utilization  [Outpatient  Drug  Utilization 


HEDIS  1999  Specifications 
(changes  from  HEDIS  3.0) 

requirements  will  be  modified. 
Medical  record  documentation 
requirements  will  be  modified. 


Medical  record  documentation 
requirements  will  be  modified. 
Procedures  modified. 


Clarification  on  how  to  identify 
and  count  certain  ambulatory 
services. 


Language  and  Table  51 
clarified. 


Measure  retired. 


Measure  retired. 
Language  clarified. 
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HEDIS  3.0/1998 
Specifications 

HEDIS  1999  Specifications 

HEDIS  1999  Specifications 
(changes  from  HEDIS  3.0) 

COST  OF  CARE 

COST  OF  CARE 

Rate  Trends 

Rate  Trends 

Language  clarified. 

High-Occurrence/High-Cost 

DRGs 

High-Occurrence/High-Cost 

DRGs 

HEALTH  PLAN 

DESCRIPTIVE 

INFORMATION 

HEALTH  PLAN 

DESCRIPTIVE 

INFORMATION 

Board  Certification/Residency 
Completion 

Board  Certification/Residency 
Completion 

Provider  Compensation 

Provider  Compensation 

Arrangements  with  Public 
Health,  Educational  and  Social 
Service  Organizations 

Arrangements  with  Public 
Health,  Educational  and  Social 
Service  Organizations 

Total  Enrollment 

Total  Enrollment 

Enrollment  by  Payer  (Member 
Years/Months) 

Enrollment  by  Payer  (Member 
Years/Months) 

Unduplicated  Count  of 
Medicaid  Members 

Unduplicated  Count  of 
Medicaid  Members 

Cultural  Diversity  of  Medicaid 
Membership 

Cultural  Diversity  of  Medicaid 
Membership 

Weeks  of  Pregnancy  at  Time 
of  Enrollment  in  the  Health 
Plan 

Weeks  of  Pregnancy  at  Time  of 
Enrollment  in  the  Health  Plan 

Note.  From  NCQA  HEDIS  1999  Draft  Document,  by  the  National  Committee  on  Quality 
Assurance,  1998,  [Text  file].  URL  ftp://info.ncqa.org/ftp/heddraft.doc 
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Appendix  H 

Health  Care  Advisory  Board  (HCAB)  Study  Indicator  Specifications  (1996) 

PERFORMANCE  INDICATOR 
Appropriateness  Indicators 

Transurethral  Resection  of  the  Prostate  (TURP)/100  Discharges 

Total  number  Total  Joint  Replacements 

Percentage  complications,  Laparoscopic  Cholecystectomies 

Total  Open  Cholecystectomies 

Hysterectomies 

Cardiac  Catheterization 

Total  Number  Back  Procedures  w/cc  -  DRG  214 
Total  Number  Back  Procedures  w/o  cc  -  DRG  215 
Total  Number  of  Backs 

Total  Number  DRG  214  &  215/100  Discharges 

Community  Benefit  Indicators 
Percent  Occupancy  Licensed  Beds 
Percent  Occupancy  Staffed  Beds 
Percent  Live  Births<2500  grams 

Comorbidities 

Percent  Discharges  with  Diabetes  Mellitus 

Percent  Discharges  with  Congestive  Heart  Failure  (CHF) 

Percent  Discharges  with  Cancer 

Percent  Discharges  with  Myocardial  Infarction  (MI) 

Percent  Discharges  with  Chronic  Renal  Failure 
Percent  Discharges  with  Pneumonia  (COPD) 

Efficiency  Indicators 

Acute  Care  Length  of  Stay  (LOS) 

Acute  Care  Discharge 
Medicare  LOS 

Finance  Indicators 

Percent  Medicare  Discharges 

Percent  Medicaid  Discharges 

Percent  of  Total  Discharges  Medicare/Medicaid 

Percent  Managed/Contracted  Care  Discharges 

Maternal/Child  Care  Indicators 

Average  Maternal  LOS 
Average  Maternal  Total  Charges 
Number  of  Deliveries 

Percent  Primary  Cesarean  Sections  (C-Sections) 


Performance  Measurement 


67 


Percent  Repeat  C-Sections 

Percent  Vaginal  Birth  After  Cesareans  (VBACs) 

Total  C-Section  Rate 
Percent  Neonatality  Mortality 
Percent  Birth  Complications 

Nursing  Staffing  Indicators 

Nursing  Staffing  Ratios-%  RN,  %  LPN,  %  Licensed,  %  Unlicensed 
Ancillary  Cost 
Nursing  Turnover  Rate 

Outcome  Indicators 

Case  Mix,  All  Patients 

Number  of  Acute  Mis;  DRGs  121, 122,  123 

Acute  MI  Mortality  Rate 

Acute  MI  ALOS 

Acute  MI  Average  Total  Charges 

Number  of  Coronary  Artery  Bypass  Grafts  (CABGs);  DRGs  106  &  107 
CABG  ALOS 

CABG  Average  Total  Charges 
Percent  CABGs  With  Cath 

Total  Percutaneous  Transluminal  Angioplasty  (PTCAs)  cases 

Number  of  Cases  Pneumonia;  DRGs  89  &  90 

Pneumonia  Mortality  Rate 

Pneumonia  ALOS 

Pneumonia  Average  Total  Charges 

Number  of  Cases  Respiratory  Infections  with  Complication/  Comorbidity;  DRG  79 

Respiratory  Infection  Mortality  Rate 

Respiratory  Infection  ALOS 

Respiratory  Infection  Average  Total  Charges 

Number  of  Cases  Cerbrovascular  Accident  (CVA)  Except  TIA  (Transient  ischemia  attacks); 
DRG  014 

CVA  Mortality  Rate 
CVA  ALOS 

CVA  Average  Total  Charges 
Number  of  Cases  CHF;  DRG  127 
CHF  Mortality  Rate 
CHF  ALOS 

CHF  Average  Total  Charges 

Number  Total  Joint  Replacements  (TJR)  Lower  Extremities 
TJR  Mortality  Rate 
TJR  ALOS 

TJR  Average  Total  Charges 
TJR  Average  Severity 

Number  of  Acute  Myocardial  Infarctions  (AMI)  by  ICD-9  Codes  410.0-410.9 
AMI  Mortality  Rate  by  ICD-9  Codes  410.0-410.9 
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Prevention  Indicators 

Emergency  Room  (ER)  Visits/100,000  Population 
Percent  ER  Stays  >  6  Hours 

Percent  Returns  to  the  Emergency  Department  w/in  72  Hours 
Readmissions  w/in  31  Days  of  Discharge 
Percent  of  ER  Patient  Admitted  to  the  Hospital 
Rate  of  ER  Patients  Admitted  to  the  Hospital 

Rate  of  Immunization  Preventable  Pneumonia  and  Influenza  in  the  Elderly 
Pediatric  Asthma  Discharge  Rate 

Quality  Indicators 

Adverse  Drug  Reaction  Rate 
Inpatient  Mortality  Rate 
Perioperative  Mortality  Rate 


Risk  Indicators 
Number  of  Maternal  Transfers 
Falls/100  Patient  Days 
Medication  Discrepancies 

Biliary  Tract  Injury  w/  Laparoscopic  Cholecystectomy  Rate 

Customer  Satisfaction  Indicators 
Hospital  Care  and  Services 
Admission  Process 
Housekeeping  Services 
Nursing  Staff 
Hospital  Employees 
Food 

Physician  Care 

Emotional  Support 

Respect  for  Privacy  and  Dignity 

Discharge  Process 

Rate  Health  Now 

Perception  of  Outcome 

Severity  Indicators 
ER  as  Admit  Source 
Case  Mix  All  Patients 
Percent  Comorbidities 

Note.  From  “Hospital-Wide  Quality  Data  Indicators,”  by  the  Health  Care  Advisory  Board,  1996, 
The  Advisory  Board  Company. 
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Appendix  I 

Health  Care  Advisory  Board  (HCAB)  Study  Indicator  Specifications  (1995) 

PERFORMANCE  MEASURES 
General  Indicators 

Average  hospital  census 
Average  length  of  stay 

Changes  in  the  number  of  lab  tests  per  patient 

Inpatient  and  outpatient  activity  trends 

Memberships — number  of  subscribers 

Number  of  admissions 

Number  of  deliveries 

Number  of  emergency  room  visits 

Plans  for  new  facilities  and  equipment 

Reassignment  pool  progress  for  displaced  employees 

Update  on  advertising  campaign 

Updates  on  case  management  and  utilization  review 

Updates  on  mental  health  and  rehabilitation  strategic  plans 

Voluntary  member  renewal  rate 

Community  Service 

County  date  rates  per  100,000  population 

Infant  mortality— deaths  per  1,000  live  births 

Number  of  calls  received  by  physician  referral  and  health  information  resource 
Percent  immunization  rate  by  age  2 

Prenatal  care — percent  of  pregnant  women  beginning  care  in  first  trimester 

System-wide  contributions  to  community  benefits 

Uncompensated  care — percent  of  total  revenue 

Uninsured — percent  of  total  population  without  health  insurance 

Updates  on  charitable  campaigns 

Financial  Performance 

Accounts  receivable — net  days  outstanding  by  quarter 
Administrative  expenses — per  member  month 
Ambulatory  surgery  discharges 
Anecdotes  on  specific  cost  saving  measures 
Average  monthly  premium  by  HMO 
Consolidated  excess  margin 
Consolidated  expenses — percent  change 
Days  of  cash  on  hand 

Expenses — per  member  month/per  member  per  month 
Fringe  benefits — percent  change 
HMO  enrollees  for  area  HMOs 

Hospital  non-Medicare  patient  days  per  1 ,000  members 

Hospitals’  unreimbursed  charges  related  to  Medicare  and  Medicaid  discounts,  indigent  care,  and 
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bad  debt 
Net  income 

Occupancy — retirement  center,  elderhouse,  and  health  center 
Other  medical  expenses — per  member  month 
Patient/customer  service  performance  measures 
Payor  mix — inpatient  activity 
Payor  mix — outpatient  activity 

Physician/Professional  service — expenses  per  member  month 

Projected  income  from  operations  for  fiscal  year  and  explanations  for  changes  from  previous 
years 

Reduction  of  controllable  expenses  per  adjusted  admission 

Salaries — percent  change 

Share  of  patient  discharges  in  12  county  market 

State  employee  bids — family  coverage 

Supplies  and  other  expenses — percent  change 

Quality  Statistics 

Changes  in  Cesarean  section  rates 

Employee  recognition  programs 

Highlights  of  achievement  and  trends  in  different  departments 
Highlights  of  outstanding  achievements  in  patient  satisfaction  reports 
Information  on  departmental  consolidation,  integration,  and  management 
Mammography  screening — percent  of  women  between  50  and  65  years  of  age  who  have  been 
screened  (year  to  date) 

Number  of  new  quality  improvement  teams  and  amount  of  associated  annual  savings  and  cost 
avoidance 

Number  of  quality  teams 

Prenatal  care — percent  of  women  who  received  prenatal  care  in  the  first  trimester 

Quality  and  cost  savings  updates 

Updates  on  progress  in  patient-focused  care 

Customer-Related  Indicators 

Customer  satisfaction  reports 
Medication  errors  per  10,000  doses — monthly 
Percent  of  members  who  maintain  primary  care  manager 
Satisfaction  with  ability  to  schedule  visits 

Note.  From  “Internal  Performance  Indicators,”  by  the  Health  Care  Advisory  Board,  1995,  The 
Advisory  Board  Company. 
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Appendix  J 


Data  Codebook 


Variable 

Number 

Code 

Description 

Entry 

Value 

Software 

Variable  Name 

1 

Initiative  Name 

INITNAM 

HEDIS  3.0/99 

1 

CRISP 

2 

CHAMP 

3 

AFMS  PMT 

4 

MHS  PRC 

5 

HCAB  Study 

6 

2 

Initiative  Type 

INITTYP 

military 

1 

civilian 

2 

3 

Indicator  Category 

INDICAT 

financial  performance 

1 

customer  satisfaction 

2 

quality — general 

3 

quality — preventive 

4 

quality — clinical  outcome 

5 

quality — disease  mgt 

6 

quality — access  to  care 

7 

community  service 

8 

utilization 

9 

enrollment 

10 

population 

11 

other 

12 

4 

Indicator  Criteria 

INDICRIT 

process 

1 

outcome 

2 

structure 

3 
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